MARATONA BEHIND THE CODE 2020

DESAFIO 4: ALGAR

Testes com modelo LightGBM

title

Parte 01. Importando as bibliotecas

In [1]:
# Pacotes padrao
import pandas as pd
import numpy as np
import pandas_profiling as pp
import seaborn as sns
import matplotlib.pylab as plt
%matplotlib inline

# Pacote do sklearn
from xgboost import XGBClassifier
import xgboost as xgb
#import lightgbm as lgbm
#import optuna.integration.lightgbm as lgb2
from sklearn.model_selection import train_test_split, KFold
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report, precision_score
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import StandardScaler, MinMaxScaler

# Pacote imblearn
import imblearn
from imblearn.over_sampling import SMOTE

# Pacote LOFO para Feature Importance
from lofo import LOFOImportance, Dataset, plot_importance

# Pacote para hyperparametros
import optuna
import functools

# Desativando warnings no jupyter
import warnings
warnings.filterwarnings('ignore')

# Configurando o output no Jupyter
pd.options.display.max_seq_items = 8000
pd.options.display.max_columns = 1000
pd.options.display.max_rows    = 1000
pd.set_option('display.max_columns', None)

Using TensorFlow backend.
In [2]:
# Versões dos pacotes usados neste jupyter notebook
%reload_ext watermark
%watermark -a "IBM Desafio 4 - Algar" --iversions

numpy            1.18.2
seaborn          0.10.0
matplotlib.pylab 1.18.2
lightgbm         2.3.1
pandas_profiling 2.8.0
pandas           0.25.3
xgboost          1.0.2
optuna           1.3.0
imblearn         0.6.2
IBM Desafio 4 - Algar

Parte 02. Carregando o Dataset

In [3]:
# Carregando o dataset da competição
dados = pd.read_csv('../data/algar-dataset-treino.csv')

# Visualizando os primeiros registros
dados.head()
Out[3]:
Idade Local de trabalho Pontuação teste Departmento Distancia casa-trabalho Educacao Area Possui carro Subordinado Satisfação com o ambiente no emprego atual Genero Horas voluntariado Envolvimento com trabalho Posicao Cargo Satisfação com emprego Estado civil Renda Bonus de performance Quantidade de empresas que trabalho Maior de idade Necessita de hora extra Aumento de salario% Performance na entrevista Satisfação com a relação Horas de trabalho padrão Beneficios Anos de experiencia Horas de treinamento ultimo ano Estilo de vida Anos na última empresa Anos na posição atual Anos desde última promoção Anos com a mesma gerência Contratar
0 49 Cliente 279 Engenharia 8 Médio completo Ciências das natureza 1 2 3 M 61 2 2 Engenheiro 2 Casado 5130 24907 1 1 Não 23 4 4 80 1 10 3 3 10 7 1 7 Não
1 33 Misto 1392 Engenharia 3 Superior incompleto - cursando Ciências das natureza 1 5 4 F 56 3 1 Engenheiro 3 Casado 2909 23159 1 1 Sim 11 3 3 80 0 8 3 3 8 7 3 0 Não
2 27 Cliente 591 Engenharia 2 Médio completo Medicina 1 7 1 M 40 3 1 Tecnico 2 Casado 3468 16632 9 1 Não 12 3 4 80 1 6 3 3 2 2 2 2 Não
3 32 Misto 1005 Engenharia 2 Superior incompleto Ciências das natureza 1 8 4 M 79 3 1 Tecnico 4 Solteiro 3068 11864 0 1 Não 13 3 3 80 0 8 2 2 7 7 3 6 Não
4 59 Misto 1324 Engenharia 3 Superior completo Medicina 1 10 3 F 81 4 1 Tecnico 1 Casado 2670 9964 4 1 Sim 20 4 1 80 3 12 3 2 1 0 0 0 Não

Parte 03. EDA (Análise Exploratória dos Dados)

Nesta fase estou usando a biblioteca Pandas Profile pois automatiza o processo e nos traz uma visão geral do dataset Para mais detalhes, segue link da documentação: https://pandas-profiling.github.io/pandas-profiling/docs/master/index.html

Dicionário de dados

  • Idade - XXXX
  • Local de trabalho - XXXX
  • Pontuação teste - XXXX
  • Departmento - XXXX
  • Distancia casa-trabalho - XXXX
  • Educacao - XXXX
  • Area - XXXX
  • Possui carro - XXXX
  • Subordinado - XXXX
  • Satisfação com o ambiente no emprego atual - XXXX
  • Genero - XXXX
  • Horas voluntariado - XXXX
  • Envolvimento com trabalho - XXXX
  • Posicao - XXXX
  • Cargo - XXXX
  • Satisfação com emprego - XXXX
  • Estado civil - XXXX
  • Renda - XXXX
  • Bonus de performance - XXXX
  • Quantidade de empresas que trabalho - XXXX
  • Maior de idade - XXXX
  • Necessita de hora extra - XXXX
  • Aumento de salario% - XXXX
  • Performance na entrevista - XXXX
  • Satisfação com a relação - XXXX
  • Horas de trabalho padrão - XXXX
  • Beneficios - XXXX
  • Anos de experiencia - XXXX
  • Horas de treinamento ultimo ano - XXXX
  • Estilo de vida - XXXX
  • Anos na última empresa - XXXX
  • Anos na posição atual - XXXX
  • Anos desde última promoção - XXXX
  • Anos com a mesma gerência - XXXX

A variável-alvo é:

  • Contratar - uma string que indica duas possibilidades:
    • "Sim" - Para contratar o candidato
    • "Não" - Para não contratar o candidato
In [4]:
# Cria o objeto profile
profile = pp.ProfileReport(dados, title="Maratona IBM - Desafio 04", explorative=True)
profile.to_file("Desafio-4-Report.html")

# Executa o relatório e mostra no Jupyter
profile










    






 
 















    















    






 
 















    















    






 
 















    















    
















    
Out[4]:





























Observações:
    

  • O dataset contem: 
    • 






- Número de features: 35
- Números de registros: 1370
- Dados missing: 0
- Dados duplicados: 0
- Features categóricas: 17
- Features numéricas: 16
- Features booleanas: 2















Parte 04. Feature Engineering
Nesta fase estarei trabalhando na manipulação dos dados
Importante realizar o mínimo de alterações para não descaracterizar as informações
Vou remover algumas colunas que identifiquei na análise exploratória e criar outras colunas que julgo interessante

















Primeiras análises


    

  • Possui carro é uma constante com todos os valores igual a 1
    • 

  • Maior de idade é uma constante com todos os valores igual a 1
    • 

  • Horas de trabalho padrão é uma constante constante com todos os valores igual a 80
    • 

  • Renda é altamente correlacionada com Posicao
    • 

  • Cargo é altamente correlacionada com Departamento
    • 

  • Subordinado só tem valores únicos (identificador único)
    • 



O que fazer?


    

  • Vou descartar todas as features constantes e identificadores
    • 














In [5]:



    


# Remove algumas features
dados.drop(['Possui carro','Maior de idade','Horas de trabalho padrão','Subordinado'], axis = 1, inplace = True)



    















Transformação das Features Categoricas Ordinais












In [6]:



    


dados['Contratar'] = dados['Contratar'].map({'Não': 0, 'Sim': 1})
dados['Educacao']  = dados['Educacao'].map({'Médio completo': 0, 
                                            'Superior incompleto': 1,
                                            'Superior incompleto - cursando': 2,
                                            'Superior completo': 3,
                                            'Pós-gradução': 4})



    













In [7]:



    


# Separar as features categoricas nominais
non_features = ['Local de trabalho','Departmento','Area','Genero','Cargo','Estado civil','Necessita de hora extra']

le = LabelEncoder()

for features in non_features:
    fe_labels = le.fit_transform(dados[features])
    dados[features] = fe_labels
    fe_mappings = {index: label for index, label in enumerate(le.classes_)}
    print(fe_mappings)



    















    






{0: 'Cliente', 1: 'Escritório', 2: 'Misto'}
{0: 'Engenharia', 1: 'RH', 2: 'Vendas'}
{0: 'Ciências das natureza', 1: 'Ciências humanas', 2: 'Faculdade Técnica', 3: 'Marketing', 4: 'Medicina', 5: 'Outros'}
{0: 'F', 1: 'M'}
{0: 'Analista', 1: 'Assistente', 2: 'Diretor', 3: 'Engenheiro', 4: 'Gerente', 5: 'Supervisor', 6: 'Tecnico', 7: 'Vendedo senior', 8: 'Vendedor junior'}
{0: 'Casado', 1: 'Divorciado', 2: 'Solteiro'}
{0: 'Não', 1: 'Sim'}



















In [8]:



    


dados.columns = dados.columns.str.replace(' ', '_')
dados.columns = dados.columns.str.normalize('NFKD').str.encode('ascii', errors='ignore').str.decode('utf-8')



    













In [9]:



    


dados.head()



    















    
Out[9]:











  
    

      
      Idade
      Local_de_trabalho
      Pontuacao_teste
      Departmento
      Distancia_casa-trabalho
      Educacao
      Area
      Satisfacao_com_o_ambiente_no_emprego_atual
      Genero
      Horas_voluntariado
      Envolvimento_com_trabalho
      Posicao
      Cargo
      Satisfacao_com_emprego
      Estado_civil
      Renda
      Bonus_de_performance
      Quantidade_de_empresas_que_trabalho
      Necessita_de_hora_extra
      Aumento_de_salario%
      Performance_na_entrevista
      Satisfacao_com_a_relacao
      Beneficios
      Anos_de_experiencia
      Horas_de_treinamento_ultimo_ano
      Estilo_de_vida
      Anos_na_ultima_empresa
      Anos_na_posicao_atual
      Anos_desde_ultima_promocao
      Anos_com_a_mesma_gerencia
      Contratar
    

  
  
    

      0
      49
      0
      279
      0
      8
      0
      0
      3
      1
      61
      2
      2
      3
      2
      0
      5130
      24907
      1
      0
      23
      4
      4
      1
      10
      3
      3
      10
      7
      1
      7
      0
    

    

      1
      33
      2
      1392
      0
      3
      2
      0
      4
      0
      56
      3
      1
      3
      3
      0
      2909
      23159
      1
      1
      11
      3
      3
      0
      8
      3
      3
      8
      7
      3
      0
      0
    

    

      2
      27
      0
      591
      0
      2
      0
      4
      1
      1
      40
      3
      1
      6
      2
      0
      3468
      16632
      9
      0
      12
      3
      4
      1
      6
      3
      3
      2
      2
      2
      2
      0
    

    

      3
      32
      2
      1005
      0
      2
      1
      0
      4
      1
      79
      3
      1
      6
      4
      2
      3068
      11864
      0
      0
      13
      3
      3
      0
      8
      2
      2
      7
      7
      3
      6
      0
    

    

      4
      59
      2
      1324
      0
      3
      3
      4
      3
      0
      81
      4
      1
      6
      1
      0
      2670
      9964
      4
      1
      20
      4
      1
      3
      12
      3
      2
      1
      0
      0
      0
      0
    

  
























Parte 05. Feature Importance
Nesta fase estarei verificando quais são as features de maior importancia para as previsoes do modelo















In [10]:



    


# define the validation scheme
cv = KFold(n_splits=5, shuffle=False, random_state=42)

# define the binary target and the features
dataset = Dataset(df=dados, target="Contratar", features=[col for col in dados.columns if col != 'Contratar'])

# define the validation scheme and scorer. The default model is LightGBM
lofo_imp = LOFOImportance(dataset, cv=cv, scoring="neg_mean_absolute_error")

# get the mean and standard deviation of the importances in pandas format
importance_df = lofo_imp.get_importance()

# plot the means and standard deviations of the importances
plot_importance(importance_df, figsize=(12, 20))



    















    






 
 















    















    




























Parte 06. Modelagem Preditiva
Nesta fase estarei construindo os modelos preditivos
Vou utilizar apenas o algoritmo XGB Classifier neste notebook

















XGBoost versão 01
    

  • Rodando a primeira versao com todas as features criadas
    • 

  • Usando train_test_split com 20% dos dados para testes
    • 

  • Sem aplicar nenhuma técnica de balanceamento
    • 
















In [11]:



    


# Vamos armazenar 'Contratar' no rótulo (y) e o restante das colunas em X
X = dados.drop(['Contratar'], axis = 1)
y = dados['Contratar']

# Aplicando a mesma escala nos dados
X = MinMaxScaler().fit_transform(X)

# Separação dos dados em um conjunto de treino e um conjunto de teste
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Criando o modelo
modelo_xgb_v1 = XGBClassifier()

# Treinando o modelo
modelo_xgb_v1.fit(X_train, y_train)

# Fazendo previsões
xgb_y_pred = modelo_xgb_v1.predict(X_test)
previsoes = [round(value) for value in xgb_y_pred]

# Avaliando as previsões
xgb_score_v1 = accuracy_score(y_test, previsoes)
print("Acurácia: %.2f%%" % (xgb_score_v1 * 100.0))

# Cria a matriz de confusão
conf_matriz_xgb = confusion_matrix(y_test, xgb_y_pred)
    
# Calcula especificidade e sensibilidade
speci_xgb = conf_matriz_xgb[0,0] / (conf_matriz_xgb[0,0] + conf_matriz_xgb[0,1])
sensi_xgb = conf_matriz_xgb[1,1] / (conf_matriz_xgb[1,0] + conf_matriz_xgb[1,1])

# Print
print('Sensibilidade :', sensi_xgb)
print('Especificidade :', speci_xgb)
print('\n')

# Matriz de Confusão gráfica
sns.set(rc={'figure.figsize':(6, 6)})
sns.heatmap(conf_matriz_xgb, 
            annot = True, 
            fmt = ".0f", 
            linewidths = .5, 
            square = True, 
            cmap = 'RdBu_r')

# Labels e Títulos
plt.ylabel('Label Verdadeiro')
plt.xlabel('Label Previsto')
plt.title('Acurácia: {:.2f}'.format(xgb_score_v1), size = 15)

# Relatório de Classificação
print(classification_report(y_test, xgb_y_pred))



    















    






Acurácia: 89.78%
Sensibilidade : 0.35294117647058826
Especificidade : 0.975


              precision    recall  f1-score   support

           0       0.91      0.97      0.94       240
           1       0.67      0.35      0.46        34

    accuracy                           0.90       274
   macro avg       0.79      0.66      0.70       274
weighted avg       0.88      0.90      0.88       274











    




























XGB versao 02
    

  • Rodando a segunda versao com todas as features
    • 

  • Usando train_test_split com 20% dos dados para testes
    • 

  • Aplicando técnica de balanceamento
    • 
















In [12]:



    


# Vamos armazenar 'Contratar' no rótulo (y) e o restante das colunas em X
X = dados.drop(['Contratar'], axis = 1)
y = dados['Contratar']

# Aplicando a funcao SMOTE
sm = SMOTE(random_state=0)
X, y = sm.fit_sample(X, y)

# Aplicando a mesma escala nos dados
X = MinMaxScaler().fit_transform(X)

# Separação dos dados em um conjunto de treino e um conjunto de teste
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Criando o modelo
modelo_xgb_v2 = XGBClassifier()

# Treinando o modelo
modelo_xgb_v2.fit(X_train, y_train)

# Fazendo previsões
xgb_y_pred = modelo_xgb_v2.predict(X_test)
previsoes = [round(value) for value in xgb_y_pred]

# Avaliando as previsões
xgb_score_v2 = accuracy_score(y_test, previsoes)
print("Acurácia: %.2f%%" % (xgb_score_v2 * 100.0))

# Cria a matriz de confusão
conf_matriz_xgb = confusion_matrix(y_test, xgb_y_pred)
    
# Calcula especificidade e sensibilidade
speci_xgb = conf_matriz_xgb[0,0] / (conf_matriz_xgb[0,0] + conf_matriz_xgb[0,1])
sensi_xgb = conf_matriz_xgb[1,1] / (conf_matriz_xgb[1,0] + conf_matriz_xgb[1,1])

# Print
print('Sensibilidade :', sensi_xgb)
print('Especificidade :', speci_xgb)
print('\n')

# Matriz de Confusão gráfica
sns.set(rc={'figure.figsize':(6, 6)})
sns.heatmap(conf_matriz_xgb, 
            annot = True, 
            fmt = ".0f", 
            linewidths = .5, 
            square = True, 
            cmap = 'RdBu_r')

# Labels e Títulos
plt.ylabel('Label Verdadeiro')
plt.xlabel('Label Previsto')
plt.title('Acurácia: {:.2f}'.format(xgb_score_v2), size = 15)

# Relatório de Classificação
print(classification_report(y_test, xgb_y_pred))



    















    






Acurácia: 93.01%
Sensibilidade : 0.9380530973451328
Especificidade : 0.9227642276422764


              precision    recall  f1-score   support

           0       0.94      0.92      0.93       246
           1       0.92      0.94      0.93       226

    accuracy                           0.93       472
   macro avg       0.93      0.93      0.93       472
weighted avg       0.93      0.93      0.93       472











    




























XGB versão 03
    

  • Rodando a terceira versao com todas as features
    • 

  • Usando train_test_split com 20% dos dados para testes
    • 

  • Aplicando técnica de balanceamento
    • 

  • Usando otimização de hyperparametros com Optuna
    • 
















In [13]:



    


def opt(X_train, y_train, X_test, y_test, trial):
    
    n_estimators     = trial.suggest_int('n_estimators', 0, 2000)
    max_depth        = trial.suggest_int('max_depth', 1, 20)
    min_child_weight = trial.suggest_int('min_child_weight', 1, 20)
    learning_rate    = trial.suggest_loguniform("learning_rate", 1e-8, 1.0)
    scale_pos_weight = trial.suggest_int('scale_pos_weight', 1, 100)
    subsample        = trial.suggest_discrete_uniform('subsample', 0.5, 0.9, 0.1)
    colsample_bytree = trial.suggest_discrete_uniform('colsample_bytree', 0.5, 0.9, 0.1)
    l1               = trial.suggest_loguniform('l1', 1e-8, 1.0)
    optimizer        = trial.suggest_categorical('optimizer', ['gbtree', 'gblinear', 'dart'])

    xgboost_tuna = xgb.XGBClassifier(
            random_state     =  42, 
            n_estimators     = n_estimators,
            max_depth        = max_depth,
            min_child_weight = min_child_weight,
            learning_rate    = learning_rate,
            scale_pos_weight = scale_pos_weight,
            subsample        = subsample,
            colsample_bytree = colsample_bytree,
            l1               = l1,
            optimizer        = optimizer
    )
    
    xgboost_tuna.fit(X_train, y_train)
    tuna_pred_test = xgboost_tuna.predict(X_test)
    
    return (1.0 - (accuracy_score(y_test, tuna_pred_test)))



    













In [14]:



    


# Vamos armazenar 'Contratar' no rótulo (y) e o restante das colunas em X
X = dados.drop(['Contratar'], axis = 1)
y = dados['Contratar']

# Aplicando a funcao SMOTE
sm = SMOTE(random_state=0)
X, y = sm.fit_sample(X, y)

# Aplicando a mesma escala nos dados
X = MinMaxScaler().fit_transform(X)

# Separação dos dados em um conjunto de treino e um conjunto de teste
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)


study = optuna.create_study()
study.optimize(functools.partial(opt, X_train, y_train, X_test, y_test), n_trials=100)



    















    






[I 2020-08-24 17:02:25,050] Finished trial#0 with value: 0.1313559322033898 with parameters: {'n_estimators': 111, 'max_depth': 11, 'min_child_weight': 19, 'learning_rate': 0.36681769044706813, 'scale_pos_weight': 46, 'subsample': 0.9, 'colsample_bytree': 0.9, 'l1': 0.0013642612659990627, 'optimizer': 'gblinear'}. Best is trial#0 with value: 0.1313559322033898.
[I 2020-08-24 17:02:39,142] Finished trial#1 with value: 0.1271186440677966 with parameters: {'n_estimators': 1833, 'max_depth': 14, 'min_child_weight': 1, 'learning_rate': 0.007388152696498991, 'scale_pos_weight': 68, 'subsample': 0.5, 'colsample_bytree': 0.7, 'l1': 3.037061504805588e-07, 'optimizer': 'gblinear'}. Best is trial#1 with value: 0.1271186440677966.
[I 2020-08-24 17:02:46,117] Finished trial#2 with value: 0.2902542372881356 with parameters: {'n_estimators': 1647, 'max_depth': 6, 'min_child_weight': 6, 'learning_rate': 3.4734732554581507e-06, 'scale_pos_weight': 8, 'subsample': 0.9, 'colsample_bytree': 0.9, 'l1': 0.01838894364857511, 'optimizer': 'gbtree'}. Best is trial#1 with value: 0.1271186440677966.
[I 2020-08-24 17:02:47,947] Finished trial#3 with value: 0.521186440677966 with parameters: {'n_estimators': 440, 'max_depth': 9, 'min_child_weight': 20, 'learning_rate': 2.9628164024880946e-05, 'scale_pos_weight': 88, 'subsample': 0.9, 'colsample_bytree': 0.6, 'l1': 2.59908305663416e-07, 'optimizer': 'dart'}. Best is trial#1 with value: 0.1271186440677966.
[I 2020-08-24 17:02:48,471] Finished trial#4 with value: 0.2542372881355932 with parameters: {'n_estimators': 321, 'max_depth': 1, 'min_child_weight': 15, 'learning_rate': 0.22925345935088884, 'scale_pos_weight': 10, 'subsample': 0.9, 'colsample_bytree': 0.6, 'l1': 2.609351981300066e-06, 'optimizer': 'dart'}. Best is trial#1 with value: 0.1271186440677966.
[I 2020-08-24 17:02:52,333] Finished trial#5 with value: 0.09745762711864403 with parameters: {'n_estimators': 1725, 'max_depth': 8, 'min_child_weight': 5, 'learning_rate': 0.24696857178704873, 'scale_pos_weight': 82, 'subsample': 0.5, 'colsample_bytree': 0.7, 'l1': 6.622038905678027e-05, 'optimizer': 'gbtree'}. Best is trial#5 with value: 0.09745762711864403.
[I 2020-08-24 17:03:00,041] Finished trial#6 with value: 0.3326271186440678 with parameters: {'n_estimators': 975, 'max_depth': 19, 'min_child_weight': 4, 'learning_rate': 4.094254048371183e-06, 'scale_pos_weight': 73, 'subsample': 0.7, 'colsample_bytree': 0.8, 'l1': 0.0009565088833995815, 'optimizer': 'dart'}. Best is trial#5 with value: 0.09745762711864403.
[I 2020-08-24 17:03:00,952] Finished trial#7 with value: 0.521186440677966 with parameters: {'n_estimators': 959, 'max_depth': 2, 'min_child_weight': 15, 'learning_rate': 2.7145200758559294e-08, 'scale_pos_weight': 75, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 0.3945093651987328, 'optimizer': 'dart'}. Best is trial#5 with value: 0.09745762711864403.
[I 2020-08-24 17:03:06,393] Finished trial#8 with value: 0.4915254237288136 with parameters: {'n_estimators': 1596, 'max_depth': 6, 'min_child_weight': 14, 'learning_rate': 3.624141644717069e-05, 'scale_pos_weight': 23, 'subsample': 0.8, 'colsample_bytree': 0.5, 'l1': 6.0177238599675096e-05, 'optimizer': 'gblinear'}. Best is trial#5 with value: 0.09745762711864403.
[I 2020-08-24 17:03:10,441] Finished trial#9 with value: 0.521186440677966 with parameters: {'n_estimators': 1058, 'max_depth': 5, 'min_child_weight': 14, 'learning_rate': 9.125452817826343e-08, 'scale_pos_weight': 44, 'subsample': 0.5, 'colsample_bytree': 0.5, 'l1': 0.2263502403939873, 'optimizer': 'gbtree'}. Best is trial#5 with value: 0.09745762711864403.
[I 2020-08-24 17:03:20,709] Finished trial#10 with value: 0.260593220338983 with parameters: {'n_estimators': 1387, 'max_depth': 16, 'min_child_weight': 8, 'learning_rate': 0.00434793308258917, 'scale_pos_weight': 96, 'subsample': 0.6, 'colsample_bytree': 0.7, 'l1': 1.704432268962957e-05, 'optimizer': 'gbtree'}. Best is trial#5 with value: 0.09745762711864403.
[I 2020-08-24 17:03:33,872] Finished trial#11 with value: 0.11440677966101698 with parameters: {'n_estimators': 1990, 'max_depth': 13, 'min_child_weight': 1, 'learning_rate': 0.006979626650615232, 'scale_pos_weight': 62, 'subsample': 0.5, 'colsample_bytree': 0.7, 'l1': 1.0057345125940676e-08, 'optimizer': 'gblinear'}. Best is trial#5 with value: 0.09745762711864403.
[I 2020-08-24 17:03:48,191] Finished trial#12 with value: 0.1228813559322034 with parameters: {'n_estimators': 1976, 'max_depth': 11, 'min_child_weight': 1, 'learning_rate': 0.006251712518204682, 'scale_pos_weight': 54, 'subsample': 0.6, 'colsample_bytree': 0.8, 'l1': 1.5512702897001514e-08, 'optimizer': 'gbtree'}. Best is trial#5 with value: 0.09745762711864403.
[I 2020-08-24 17:03:55,186] Finished trial#13 with value: 0.09533898305084743 with parameters: {'n_estimators': 1886, 'max_depth': 14, 'min_child_weight': 4, 'learning_rate': 0.05345301046618364, 'scale_pos_weight': 100, 'subsample': 0.5, 'colsample_bytree': 0.8, 'l1': 1.626187326402719e-08, 'optimizer': 'gblinear'}. Best is trial#13 with value: 0.09533898305084743.
[I 2020-08-24 17:03:57,237] Finished trial#14 with value: 0.10169491525423724 with parameters: {'n_estimators': 1322, 'max_depth': 20, 'min_child_weight': 5, 'learning_rate': 0.8613991695541675, 'scale_pos_weight': 100, 'subsample': 0.6, 'colsample_bytree': 0.8, 'l1': 1.6312574648729938e-07, 'optimizer': 'gblinear'}. Best is trial#13 with value: 0.09533898305084743.
[I 2020-08-24 17:04:02,707] Finished trial#15 with value: 0.11016949152542377 with parameters: {'n_estimators': 1715, 'max_depth': 16, 'min_child_weight': 10, 'learning_rate': 0.07229068388686265, 'scale_pos_weight': 87, 'subsample': 0.5, 'colsample_bytree': 0.8, 'l1': 0.000741612680602522, 'optimizer': 'gbtree'}. Best is trial#13 with value: 0.09533898305084743.
[I 2020-08-24 17:04:09,835] Finished trial#16 with value: 0.39406779661016944 with parameters: {'n_estimators': 1333, 'max_depth': 8, 'min_child_weight': 3, 'learning_rate': 0.00035974405710185916, 'scale_pos_weight': 85, 'subsample': 0.6, 'colsample_bytree': 0.6, 'l1': 4.196136479206081e-06, 'optimizer': 'gbtree'}. Best is trial#13 with value: 0.09533898305084743.
[I 2020-08-24 17:04:15,154] Finished trial#17 with value: 0.09322033898305082 with parameters: {'n_estimators': 1958, 'max_depth': 17, 'min_child_weight': 8, 'learning_rate': 0.09489109402532123, 'scale_pos_weight': 100, 'subsample': 0.5, 'colsample_bytree': 0.9, 'l1': 0.032800757883654344, 'optimizer': 'gblinear'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:04:30,997] Finished trial#18 with value: 0.4216101694915254 with parameters: {'n_estimators': 1995, 'max_depth': 18, 'min_child_weight': 8, 'learning_rate': 0.0005317250330137562, 'scale_pos_weight': 96, 'subsample': 0.6, 'colsample_bytree': 0.9, 'l1': 0.03806081336775949, 'optimizer': 'gblinear'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:04:38,349] Finished trial#19 with value: 0.10593220338983056 with parameters: {'n_estimators': 1505, 'max_depth': 16, 'min_child_weight': 11, 'learning_rate': 0.03603993375539022, 'scale_pos_weight': 34, 'subsample': 0.8, 'colsample_bytree': 0.9, 'l1': 0.008800372050776353, 'optimizer': 'gblinear'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:04:43,126] Finished trial#20 with value: 0.4703389830508474 with parameters: {'n_estimators': 702, 'max_depth': 13, 'min_child_weight': 8, 'learning_rate': 0.0008728989805891726, 'scale_pos_weight': 100, 'subsample': 0.5, 'colsample_bytree': 0.8, 'l1': 0.004576108678785992, 'optimizer': 'gblinear'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:04:48,207] Finished trial#21 with value: 0.09745762711864403 with parameters: {'n_estimators': 1833, 'max_depth': 14, 'min_child_weight': 6, 'learning_rate': 0.06930463308203928, 'scale_pos_weight': 80, 'subsample': 0.5, 'colsample_bytree': 0.7, 'l1': 0.00024152670644506213, 'optimizer': 'gblinear'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:04:54,485] Finished trial#22 with value: 0.09957627118644063 with parameters: {'n_estimators': 1838, 'max_depth': 14, 'min_child_weight': 7, 'learning_rate': 0.04700181013298609, 'scale_pos_weight': 100, 'subsample': 0.5, 'colsample_bytree': 0.8, 'l1': 0.13946515676820084, 'optimizer': 'gblinear'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:04:57,155] Finished trial#23 with value: 0.09957627118644063 with parameters: {'n_estimators': 1963, 'max_depth': 18, 'min_child_weight': 3, 'learning_rate': 0.9662121914883659, 'scale_pos_weight': 88, 'subsample': 0.6, 'colsample_bytree': 0.9, 'l1': 5.765226237944927e-05, 'optimizer': 'gbtree'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:05:04,006] Finished trial#24 with value: 0.10593220338983056 with parameters: {'n_estimators': 1828, 'max_depth': 15, 'min_child_weight': 10, 'learning_rate': 0.03690299959994155, 'scale_pos_weight': 95, 'subsample': 0.5, 'colsample_bytree': 0.8, 'l1': 0.8706013248044483, 'optimizer': 'gblinear'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:05:15,728] Finished trial#25 with value: 0.4067796610169492 with parameters: {'n_estimators': 1479, 'max_depth': 12, 'min_child_weight': 11, 'learning_rate': 0.0018438646064951397, 'scale_pos_weight': 78, 'subsample': 0.6, 'colsample_bytree': 0.7, 'l1': 0.0002336784146535775, 'optimizer': 'gblinear'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:05:18,823] Finished trial#26 with value: 0.09322033898305082 with parameters: {'n_estimators': 1175, 'max_depth': 17, 'min_child_weight': 3, 'learning_rate': 0.10337557828955495, 'scale_pos_weight': 65, 'subsample': 0.5, 'colsample_bytree': 0.6, 'l1': 1.0930080165125732e-05, 'optimizer': 'gblinear'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:05:20,571] Finished trial#27 with value: 0.10593220338983056 with parameters: {'n_estimators': 1162, 'max_depth': 18, 'min_child_weight': 3, 'learning_rate': 0.8505197561009885, 'scale_pos_weight': 60, 'subsample': 0.7, 'colsample_bytree': 0.6, 'l1': 2.2518868153953244e-06, 'optimizer': 'gblinear'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:05:24,807] Finished trial#28 with value: 0.10593220338983056 with parameters: {'n_estimators': 703, 'max_depth': 20, 'min_child_weight': 2, 'learning_rate': 0.02378177372514463, 'scale_pos_weight': 30, 'subsample': 0.5, 'colsample_bytree': 0.6, 'l1': 2.5281348143716833e-08, 'optimizer': 'gblinear'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:05:25,485] Finished trial#29 with value: 0.2055084745762712 with parameters: {'n_estimators': 28, 'max_depth': 17, 'min_child_weight': 5, 'learning_rate': 0.2192608505285061, 'scale_pos_weight': 49, 'subsample': 0.6, 'colsample_bytree': 0.9, 'l1': 6.506375477391823e-08, 'optimizer': 'gblinear'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:05:27,489] Finished trial#30 with value: 0.11016949152542377 with parameters: {'n_estimators': 675, 'max_depth': 10, 'min_child_weight': 9, 'learning_rate': 0.17877679891698295, 'scale_pos_weight': 67, 'subsample': 0.8, 'colsample_bytree': 0.9, 'l1': 1.0051060189967472e-05, 'optimizer': 'gblinear'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:05:29,924] Finished trial#31 with value: 0.09957627118644063 with parameters: {'n_estimators': 1730, 'max_depth': 8, 'min_child_weight': 5, 'learning_rate': 0.39331350851083713, 'scale_pos_weight': 91, 'subsample': 0.5, 'colsample_bytree': 0.7, 'l1': 9.386159441276598e-07, 'optimizer': 'gbtree'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:05:37,053] Finished trial#32 with value: 0.1292372881355932 with parameters: {'n_estimators': 1159, 'max_depth': 12, 'min_child_weight': 4, 'learning_rate': 0.015886974112546637, 'scale_pos_weight': 81, 'subsample': 0.5, 'colsample_bytree': 0.6, 'l1': 1.8444854269378826e-05, 'optimizer': 'gblinear'}. Best is trial#17 with value: 0.09322033898305082.
[I 2020-08-24 17:05:41,473] Finished trial#33 with value: 0.08898305084745761 with parameters: {'n_estimators': 1878, 'max_depth': 15, 'min_child_weight': 7, 'learning_rate': 0.10063199691108342, 'scale_pos_weight': 67, 'subsample': 0.5, 'colsample_bytree': 0.7, 'l1': 0.00027999668304249834, 'optimizer': 'gblinear'}. Best is trial#33 with value: 0.08898305084745761.
[I 2020-08-24 17:05:53,179] Finished trial#34 with value: 0.2860169491525424 with parameters: {'n_estimators': 1588, 'max_depth': 15, 'min_child_weight': 7, 'learning_rate': 0.002314885642586454, 'scale_pos_weight': 56, 'subsample': 0.5, 'colsample_bytree': 0.8, 'l1': 0.002886472057420454, 'optimizer': 'gblinear'}. Best is trial#33 with value: 0.08898305084745761.
[I 2020-08-24 17:06:02,343] Finished trial#35 with value: 0.17372881355932202 with parameters: {'n_estimators': 1855, 'max_depth': 17, 'min_child_weight': 12, 'learning_rate': 0.00935551034993178, 'scale_pos_weight': 69, 'subsample': 0.5, 'colsample_bytree': 0.7, 'l1': 0.05447228142059216, 'optimizer': 'gblinear'}. Best is trial#33 with value: 0.08898305084745761.
[I 2020-08-24 17:06:04,255] Finished trial#36 with value: 0.08898305084745761 with parameters: {'n_estimators': 424, 'max_depth': 15, 'min_child_weight': 7, 'learning_rate': 0.10068182098746403, 'scale_pos_weight': 43, 'subsample': 0.6, 'colsample_bytree': 0.6, 'l1': 5.020522823657583e-07, 'optimizer': 'gblinear'}. Best is trial#33 with value: 0.08898305084745761.
[I 2020-08-24 17:06:05,751] Finished trial#37 with value: 0.11440677966101698 with parameters: {'n_estimators': 217, 'max_depth': 19, 'min_child_weight': 7, 'learning_rate': 0.12912907639687526, 'scale_pos_weight': 43, 'subsample': 0.6, 'colsample_bytree': 0.6, 'l1': 7.608390379107736e-07, 'optimizer': 'gblinear'}. Best is trial#33 with value: 0.08898305084745761.
[I 2020-08-24 17:06:07,127] Finished trial#38 with value: 0.09322033898305082 with parameters: {'n_estimators': 822, 'max_depth': 17, 'min_child_weight': 9, 'learning_rate': 0.3463187490816046, 'scale_pos_weight': 34, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 5.728217349126838e-07, 'optimizer': 'dart'}. Best is trial#33 with value: 0.08898305084745761.
[I 2020-08-24 17:06:08,869] Finished trial#39 with value: 0.09322033898305082 with parameters: {'n_estimators': 570, 'max_depth': 19, 'min_child_weight': 18, 'learning_rate': 0.4254589573107078, 'scale_pos_weight': 18, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 5.343616144965187e-07, 'optimizer': 'dart'}. Best is trial#33 with value: 0.08898305084745761.
[I 2020-08-24 17:06:10,001] Finished trial#40 with value: 0.11228813559322037 with parameters: {'n_estimators': 521, 'max_depth': 19, 'min_child_weight': 20, 'learning_rate': 0.9794988976090129, 'scale_pos_weight': 16, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 8.096268470109911e-08, 'optimizer': 'dart'}. Best is trial#33 with value: 0.08898305084745761.
[I 2020-08-24 17:06:11,286] Finished trial#41 with value: 0.08686440677966101 with parameters: {'n_estimators': 809, 'max_depth': 15, 'min_child_weight': 18, 'learning_rate': 0.3741198853891014, 'scale_pos_weight': 1, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 5.084540620001771e-07, 'optimizer': 'dart'}. Best is trial#41 with value: 0.08686440677966101.
[I 2020-08-24 17:06:13,508] Finished trial#42 with value: 0.07415254237288138 with parameters: {'n_estimators': 868, 'max_depth': 15, 'min_child_weight': 17, 'learning_rate': 0.10574142043680212, 'scale_pos_weight': 1, 'subsample': 0.8, 'colsample_bytree': 0.6, 'l1': 3.5211481976903677e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:14,212] Finished trial#43 with value: 0.08898305084745761 with parameters: {'n_estimators': 441, 'max_depth': 15, 'min_child_weight': 18, 'learning_rate': 0.4253687805154992, 'scale_pos_weight': 2, 'subsample': 0.8, 'colsample_bytree': 0.5, 'l1': 2.9211308917556205e-07, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:15,737] Finished trial#44 with value: 0.11016949152542377 with parameters: {'n_estimators': 355, 'max_depth': 15, 'min_child_weight': 17, 'learning_rate': 0.014789523278636254, 'scale_pos_weight': 1, 'subsample': 0.8, 'colsample_bytree': 0.5, 'l1': 2.0791734397350813e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:19,192] Finished trial#45 with value: 0.18008474576271183 with parameters: {'n_estimators': 835, 'max_depth': 13, 'min_child_weight': 17, 'learning_rate': 3.190698270678235e-06, 'scale_pos_weight': 2, 'subsample': 0.9, 'colsample_bytree': 0.5, 'l1': 1.0707851809629353e-07, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:20,307] Finished trial#46 with value: 0.09957627118644063 with parameters: {'n_estimators': 251, 'max_depth': 12, 'min_child_weight': 18, 'learning_rate': 0.5683599839696986, 'scale_pos_weight': 8, 'subsample': 0.8, 'colsample_bytree': 0.6, 'l1': 1.8909187450874226e-07, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:22,856] Finished trial#47 with value: 0.4152542372881356 with parameters: {'n_estimators': 457, 'max_depth': 15, 'min_child_weight': 20, 'learning_rate': 9.919073178097145e-07, 'scale_pos_weight': 7, 'subsample': 0.9, 'colsample_bytree': 0.5, 'l1': 4.4234071965715184e-08, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:27,448] Finished trial#48 with value: 0.24152542372881358 with parameters: {'n_estimators': 898, 'max_depth': 10, 'min_child_weight': 16, 'learning_rate': 0.00016933569820044782, 'scale_pos_weight': 3, 'subsample': 0.8, 'colsample_bytree': 0.6, 'l1': 3.2749679297909613e-07, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:28,071] Finished trial#49 with value: 0.11440677966101698 with parameters: {'n_estimators': 144, 'max_depth': 13, 'min_child_weight': 19, 'learning_rate': 0.2633648464087737, 'scale_pos_weight': 16, 'subsample': 0.8, 'colsample_bytree': 0.5, 'l1': 4.056265277487281e-05, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:35,413] Finished trial#50 with value: 0.23305084745762716 with parameters: {'n_estimators': 1057, 'max_depth': 16, 'min_child_weight': 13, 'learning_rate': 0.003495841229954719, 'scale_pos_weight': 12, 'subsample': 0.8, 'colsample_bytree': 0.6, 'l1': 3.6118403978200213e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:36,977] Finished trial#51 with value: 0.0826271186440678 with parameters: {'n_estimators': 848, 'max_depth': 14, 'min_child_weight': 9, 'learning_rate': 0.44435350033995197, 'scale_pos_weight': 39, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 1.1074599520762664e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:39,248] Finished trial#52 with value: 0.11016949152542377 with parameters: {'n_estimators': 801, 'max_depth': 14, 'min_child_weight': 19, 'learning_rate': 0.13948576850792233, 'scale_pos_weight': 36, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 1.77924850282691e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:40,654] Finished trial#53 with value: 0.07627118644067798 with parameters: {'n_estimators': 643, 'max_depth': 15, 'min_child_weight': 6, 'learning_rate': 0.5626749105571682, 'scale_pos_weight': 25, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 6.455752182568991e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:43,153] Finished trial#54 with value: 0.08686440677966101 with parameters: {'n_estimators': 611, 'max_depth': 14, 'min_child_weight': 15, 'learning_rate': 0.648866576291438, 'scale_pos_weight': 25, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 1.1206739377786462e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:44,871] Finished trial#55 with value: 0.08686440677966101 with parameters: {'n_estimators': 601, 'max_depth': 11, 'min_child_weight': 15, 'learning_rate': 0.7838771953775528, 'scale_pos_weight': 27, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 1.3557814425178453e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:46,607] Finished trial#56 with value: 0.07627118644067798 with parameters: {'n_estimators': 628, 'max_depth': 11, 'min_child_weight': 15, 'learning_rate': 0.8281680927392756, 'scale_pos_weight': 26, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 1.2512294516568965e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:48,012] Finished trial#57 with value: 0.09533898305084743 with parameters: {'n_estimators': 579, 'max_depth': 11, 'min_child_weight': 15, 'learning_rate': 0.7219831951486672, 'scale_pos_weight': 25, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 5.5379899725692966e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:49,135] Finished trial#58 with value: 0.10381355932203384 with parameters: {'n_estimators': 624, 'max_depth': 9, 'min_child_weight': 14, 'learning_rate': 0.994322428775753, 'scale_pos_weight': 25, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 1.2234323615191117e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:51,231] Finished trial#59 with value: 0.11228813559322037 with parameters: {'n_estimators': 761, 'max_depth': 11, 'min_child_weight': 16, 'learning_rate': 0.254975499351104, 'scale_pos_weight': 39, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 5.635605240452315e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:57,786] Finished trial#60 with value: 0.5042372881355932 with parameters: {'n_estimators': 939, 'max_depth': 12, 'min_child_weight': 13, 'learning_rate': 2.44344117495811e-05, 'scale_pos_weight': 30, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 2.327415725052454e-05, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:06:58,803] Finished trial#61 with value: 0.08686440677966101 with parameters: {'n_estimators': 895, 'max_depth': 13, 'min_child_weight': 15, 'learning_rate': 0.9195542373880179, 'scale_pos_weight': 19, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 1.482551377127457e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:00,306] Finished trial#62 with value: 0.09745762711864403 with parameters: {'n_estimators': 895, 'max_depth': 13, 'min_child_weight': 16, 'learning_rate': 0.4803425504552507, 'scale_pos_weight': 20, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 8.343223367174926e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:02,416] Finished trial#63 with value: 0.08686440677966101 with parameters: {'n_estimators': 1024, 'max_depth': 14, 'min_child_weight': 17, 'learning_rate': 0.20205485642652718, 'scale_pos_weight': 12, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 3.59041266600712e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:06,382] Finished trial#64 with value: 0.10805084745762716 with parameters: {'n_estimators': 1033, 'max_depth': 14, 'min_child_weight': 17, 'learning_rate': 0.03664605151713916, 'scale_pos_weight': 13, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 2.982039010112717e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:07,415] Finished trial#65 with value: 0.09957627118644063 with parameters: {'n_estimators': 743, 'max_depth': 9, 'min_child_weight': 15, 'learning_rate': 0.980800345636752, 'scale_pos_weight': 27, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 1.2404231657033582e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:09,593] Finished trial#66 with value: 0.08686440677966101 with parameters: {'n_estimators': 1108, 'max_depth': 16, 'min_child_weight': 17, 'learning_rate': 0.16817361072300482, 'scale_pos_weight': 21, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 2.8927436906182954e-05, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:12,844] Finished trial#67 with value: 0.10593220338983056 with parameters: {'n_estimators': 1266, 'max_depth': 13, 'min_child_weight': 14, 'learning_rate': 0.053206723236740194, 'scale_pos_weight': 21, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 0.00011349104683811185, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:14,594] Finished trial#68 with value: 0.09533898305084743 with parameters: {'n_estimators': 978, 'max_depth': 12, 'min_child_weight': 16, 'learning_rate': 0.22618079478880332, 'scale_pos_weight': 8, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 1.6142157686878991e-07, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:16,386] Finished trial#69 with value: 0.17796610169491522 with parameters: {'n_estimators': 635, 'max_depth': 5, 'min_child_weight': 13, 'learning_rate': 0.020297832428897974, 'scale_pos_weight': 29, 'subsample': 0.7, 'colsample_bytree': 0.6, 'l1': 2.7610163317281023e-05, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:19,790] Finished trial#70 with value: 0.5148305084745763 with parameters: {'n_estimators': 525, 'max_depth': 10, 'min_child_weight': 12, 'learning_rate': 1.3138724451715023e-08, 'scale_pos_weight': 39, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 1.4510698046954025e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:20,839] Finished trial#71 with value: 0.0826271186440678 with parameters: {'n_estimators': 884, 'max_depth': 14, 'min_child_weight': 18, 'learning_rate': 0.5725195760140571, 'scale_pos_weight': 5, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 1.1995496022992828e-05, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:23,149] Finished trial#72 with value: 0.09533898305084743 with parameters: {'n_estimators': 1132, 'max_depth': 16, 'min_child_weight': 17, 'learning_rate': 0.1693676014666254, 'scale_pos_weight': 14, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 1.2912784367009184e-05, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:26,085] Finished trial#73 with value: 0.11016949152542377 with parameters: {'n_estimators': 1110, 'max_depth': 16, 'min_child_weight': 19, 'learning_rate': 0.060100501934620895, 'scale_pos_weight': 20, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 5.984952591809819e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:27,641] Finished trial#74 with value: 0.0826271186440678 with parameters: {'n_estimators': 888, 'max_depth': 14, 'min_child_weight': 16, 'learning_rate': 0.5075891626646222, 'scale_pos_weight': 6, 'subsample': 0.7, 'colsample_bytree': 0.5, 'l1': 3.0487060997639e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:28,666] Finished trial#75 with value: 0.09957627118644063 with parameters: {'n_estimators': 671, 'max_depth': 16, 'min_child_weight': 16, 'learning_rate': 0.5482975102068446, 'scale_pos_weight': 5, 'subsample': 0.6, 'colsample_bytree': 0.5, 'l1': 6.687010526293176e-05, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:30,193] Finished trial#76 with value: 0.07838983050847459 with parameters: {'n_estimators': 891, 'max_depth': 14, 'min_child_weight': 6, 'learning_rate': 0.2878836952768678, 'scale_pos_weight': 10, 'subsample': 0.7, 'colsample_bytree': 0.6, 'l1': 4.0481024904001246e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:31,753] Finished trial#77 with value: 0.07627118644067798 with parameters: {'n_estimators': 867, 'max_depth': 15, 'min_child_weight': 6, 'learning_rate': 0.36636817104794134, 'scale_pos_weight': 6, 'subsample': 0.8, 'colsample_bytree': 0.6, 'l1': 7.778946489764274e-06, 'optimizer': 'dart'}. Best is trial#42 with value: 0.07415254237288138.
[I 2020-08-24 17:07:34,302] Finished trial#78 with value: 0.06567796610169496 with parameters: {'n_estimators': 759, 'max_depth': 14, 'min_child_weight': 6, 'learning_rate': 0.07243530790309537, 'scale_pos_weight': 5, 'subsample': 0.8, 'colsample_bytree': 0.6, 'l1': 7.714138887917236e-06, 'optimizer': 'dart'}. Best is trial#78 with value: 0.06567796610169496.
[I 2020-08-24 17:07:37,066] Finished trial#79 with value: 0.07838983050847459 with parameters: {'n_estimators': 776, 'max_depth': 13, 'min_child_weight': 6, 'learning_rate': 0.086754480516766, 'scale_pos_weight': 10, 'subsample': 0.8, 'colsample_bytree': 0.6, 'l1': 1.4894570101808572e-05, 'optimizer': 'dart'}. Best is trial#78 with value: 0.06567796610169496.
[I 2020-08-24 17:07:40,422] Finished trial#80 with value: 0.07838983050847459 with parameters: {'n_estimators': 760, 'max_depth': 12, 'min_child_weight': 4, 'learning_rate': 0.02735270469711791, 'scale_pos_weight': 10, 'subsample': 0.8, 'colsample_bytree': 0.6, 'l1': 8.272769014650836e-06, 'optimizer': 'gbtree'}. Best is trial#78 with value: 0.06567796610169496.
[I 2020-08-24 17:07:43,732] Finished trial#81 with value: 0.07203389830508478 with parameters: {'n_estimators': 741, 'max_depth': 13, 'min_child_weight': 6, 'learning_rate': 0.07840680387166116, 'scale_pos_weight': 10, 'subsample': 0.8, 'colsample_bytree': 0.6, 'l1': 1.4555436461726294e-05, 'optimizer': 'gbtree'}. Best is trial#78 with value: 0.06567796610169496.
[I 2020-08-24 17:07:47,188] Finished trial#82 with value: 0.07838983050847459 with parameters: {'n_estimators': 739, 'max_depth': 13, 'min_child_weight': 6, 'learning_rate': 0.024186717248280434, 'scale_pos_weight': 9, 'subsample': 0.8, 'colsample_bytree': 0.6, 'l1': 8.48799270905425e-06, 'optimizer': 'dart'}. Best is trial#78 with value: 0.06567796610169496.
[I 2020-08-24 17:07:51,958] Finished trial#83 with value: 0.10593220338983056 with parameters: {'n_estimators': 742, 'max_depth': 13, 'min_child_weight': 5, 'learning_rate': 0.01220820177367654, 'scale_pos_weight': 10, 'subsample': 0.8, 'colsample_bytree': 0.6, 'l1': 1.743761533991006e-05, 'optimizer': 'gbtree'}. Best is trial#78 with value: 0.06567796610169496.
[I 2020-08-24 17:07:54,391] Finished trial#84 with value: 0.06779661016949157 with parameters: {'n_estimators': 694, 'max_depth': 12, 'min_child_weight': 6, 'learning_rate': 0.084176780387612, 'scale_pos_weight': 4, 'subsample': 0.8, 'colsample_bytree': 0.6, 'l1': 4.0798376176636395e-05, 'optimizer': 'gbtree'}. Best is trial#78 with value: 0.06567796610169496.
[I 2020-08-24 17:07:57,786] Finished trial#85 with value: 0.06355932203389836 with parameters: {'n_estimators': 688, 'max_depth': 12, 'min_child_weight': 4, 'learning_rate': 0.031110222191640887, 'scale_pos_weight': 4, 'subsample': 0.8, 'colsample_bytree': 0.6, 'l1': 0.00012453914538483387, 'optimizer': 'gbtree'}. Best is trial#85 with value: 0.06355932203389836.
[I 2020-08-24 17:08:00,593] Finished trial#86 with value: 0.06144067796610164 with parameters: {'n_estimators': 503, 'max_depth': 12, 'min_child_weight': 5, 'learning_rate': 0.0744648243444002, 'scale_pos_weight': 4, 'subsample': 0.9, 'colsample_bytree': 0.6, 'l1': 0.00013454108456274012, 'optimizer': 'gbtree'}. Best is trial#86 with value: 0.06144067796610164.
[I 2020-08-24 17:08:04,072] Finished trial#87 with value: 0.09533898305084743 with parameters: {'n_estimators': 523, 'max_depth': 12, 'min_child_weight': 2, 'learning_rate': 0.006058740860989404, 'scale_pos_weight': 5, 'subsample': 0.9, 'colsample_bytree': 0.6, 'l1': 0.00014661255571704417, 'optimizer': 'gbtree'}. Best is trial#86 with value: 0.06144067796610164.
[I 2020-08-24 17:08:06,100] Finished trial#88 with value: 0.05932203389830504 with parameters: {'n_estimators': 680, 'max_depth': 10, 'min_child_weight': 5, 'learning_rate': 0.049055230722663656, 'scale_pos_weight': 3, 'subsample': 0.8, 'colsample_bytree': 0.6, 'l1': 0.000399953832228646, 'optimizer': 'gbtree'}. Best is trial#88 with value: 0.05932203389830504.
[I 2020-08-24 17:08:08,443] Finished trial#89 with value: 0.05932203389830504 with parameters: {'n_estimators': 696, 'max_depth': 11, 'min_child_weight': 4, 'learning_rate': 0.06815704995829862, 'scale_pos_weight': 3, 'subsample': 0.8, 'colsample_bytree': 0.6, 'l1': 0.0008203293604135664, 'optimizer': 'gbtree'}. Best is trial#88 with value: 0.05932203389830504.
[I 2020-08-24 17:08:11,400] Finished trial#90 with value: 0.06355932203389836 with parameters: {'n_estimators': 674, 'max_depth': 10, 'min_child_weight': 4, 'learning_rate': 0.04279724532055313, 'scale_pos_weight': 3, 'subsample': 0.9, 'colsample_bytree': 0.7, 'l1': 0.0005234514151619235, 'optimizer': 'gbtree'}. Best is trial#88 with value: 0.05932203389830504.
[I 2020-08-24 17:08:14,007] Finished trial#91 with value: 0.06144067796610164 with parameters: {'n_estimators': 701, 'max_depth': 10, 'min_child_weight': 4, 'learning_rate': 0.04188078634933376, 'scale_pos_weight': 3, 'subsample': 0.9, 'colsample_bytree': 0.7, 'l1': 0.0006500743515262431, 'optimizer': 'gbtree'}. Best is trial#88 with value: 0.05932203389830504.
[I 2020-08-24 17:08:16,763] Finished trial#92 with value: 0.06355932203389836 with parameters: {'n_estimators': 687, 'max_depth': 8, 'min_child_weight': 4, 'learning_rate': 0.039508428343474, 'scale_pos_weight': 3, 'subsample': 0.9, 'colsample_bytree': 0.7, 'l1': 0.0003986714925258896, 'optimizer': 'gbtree'}. Best is trial#88 with value: 0.05932203389830504.
[I 2020-08-24 17:08:19,515] Finished trial#93 with value: 0.06567796610169496 with parameters: {'n_estimators': 696, 'max_depth': 7, 'min_child_weight': 4, 'learning_rate': 0.040278216768407274, 'scale_pos_weight': 3, 'subsample': 0.9, 'colsample_bytree': 0.7, 'l1': 0.0006378713081735552, 'optimizer': 'gbtree'}. Best is trial#88 with value: 0.05932203389830504.
[I 2020-08-24 17:08:22,507] Finished trial#94 with value: 0.06355932203389836 with parameters: {'n_estimators': 699, 'max_depth': 8, 'min_child_weight': 4, 'learning_rate': 0.04248870533805423, 'scale_pos_weight': 4, 'subsample': 0.9, 'colsample_bytree': 0.7, 'l1': 0.0005721357568626519, 'optimizer': 'gbtree'}. Best is trial#88 with value: 0.05932203389830504.
[I 2020-08-24 17:08:24,518] Finished trial#95 with value: 0.06355932203389836 with parameters: {'n_estimators': 502, 'max_depth': 7, 'min_child_weight': 2, 'learning_rate': 0.03701957171507942, 'scale_pos_weight': 3, 'subsample': 0.9, 'colsample_bytree': 0.7, 'l1': 0.0005207233176105686, 'optimizer': 'gbtree'}. Best is trial#88 with value: 0.05932203389830504.
[I 2020-08-24 17:08:27,298] Finished trial#96 with value: 0.05508474576271183 with parameters: {'n_estimators': 485, 'max_depth': 7, 'min_child_weight': 2, 'learning_rate': 0.04269591630105312, 'scale_pos_weight': 1, 'subsample': 0.9, 'colsample_bytree': 0.7, 'l1': 0.0005734744740979771, 'optimizer': 'gbtree'}. Best is trial#96 with value: 0.05508474576271183.
[I 2020-08-24 17:08:29,569] Finished trial#97 with value: 0.08898305084745761 with parameters: {'n_estimators': 372, 'max_depth': 7, 'min_child_weight': 2, 'learning_rate': 0.009420329482683051, 'scale_pos_weight': 1, 'subsample': 0.9, 'colsample_bytree': 0.7, 'l1': 0.0013930536232879362, 'optimizer': 'gbtree'}. Best is trial#96 with value: 0.05508474576271183.
[I 2020-08-24 17:08:32,287] Finished trial#98 with value: 0.11864406779661019 with parameters: {'n_estimators': 469, 'max_depth': 8, 'min_child_weight': 3, 'learning_rate': 0.016764239180949672, 'scale_pos_weight': 15, 'subsample': 0.9, 'colsample_bytree': 0.7, 'l1': 0.0004396306821038104, 'optimizer': 'gbtree'}. Best is trial#96 with value: 0.05508474576271183.
[I 2020-08-24 17:08:34,012] Finished trial#99 with value: 0.1228813559322034 with parameters: {'n_estimators': 494, 'max_depth': 5, 'min_child_weight': 1, 'learning_rate': 0.004934791009074267, 'scale_pos_weight': 1, 'subsample': 0.9, 'colsample_bytree': 0.7, 'l1': 0.0013842821223335156, 'optimizer': 'gbtree'}. Best is trial#96 with value: 0.05508474576271183.



















In [15]:



    


study.best_params



    















    
Out[15]:








{'n_estimators': 485,
 'max_depth': 7,
 'min_child_weight': 2,
 'learning_rate': 0.04269591630105312,
 'scale_pos_weight': 1,
 'subsample': 0.9,
 'colsample_bytree': 0.7,
 'l1': 0.0005734744740979771,
 'optimizer': 'gbtree'}



















In [16]:



    


# Criando o modelo
modelo_xgb_v3 = xgb.XGBClassifier(**study.best_params)

# Treinando o modelo
modelo_xgb_v3.fit(X_train, y_train)

# Fazendo previsões
xgb_y_pred = modelo_xgb_v3.predict(X_test)
previsoes = [round(value) for value in xgb_y_pred]

# Avaliando as previsões
xgb_score_v3 = accuracy_score(y_test, previsoes)
print("Acurácia: %.2f%%" % (xgb_score_v3 * 100.0))

# Cria a matriz de confusão
conf_matriz_xgb = confusion_matrix(y_test, xgb_y_pred)
    
# Calcula especificidade e sensibilidade
speci_xgb = conf_matriz_xgb[0,0] / (conf_matriz_xgb[0,0] + conf_matriz_xgb[0,1])
sensi_xgb = conf_matriz_xgb[1,1] / (conf_matriz_xgb[1,0] + conf_matriz_xgb[1,1])

# Print
print('Sensibilidade :', sensi_xgb)
print('Especificidade :', speci_xgb)
print('\n')

# Matriz de Confusão gráfica
sns.set(rc={'figure.figsize':(6, 6)})
sns.heatmap(conf_matriz_xgb, 
            annot = True, 
            fmt = ".0f", 
            linewidths = .5, 
            square = True, 
            cmap = 'RdBu_r')

# Labels e Títulos
plt.ylabel('Label Verdadeiro')
plt.xlabel('Label Previsto')
plt.title('Acurácia: {:.2f}'.format(xgb_score_v3), size = 15)

# Relatório de Classificação
print(classification_report(y_test, xgb_y_pred))



    















    






Acurácia: 93.86%
Sensibilidade : 0.9380530973451328
Especificidade : 0.9390243902439024


              precision    recall  f1-score   support

           0       0.94      0.94      0.94       246
           1       0.93      0.94      0.94       226

    accuracy                           0.94       472
   macro avg       0.94      0.94      0.94       472
weighted avg       0.94      0.94      0.94       472











    




























XGB versão 04
    

  • Rodando a quarta versao com todas as features
    • 

  • Usando train_test_split com 20% dos dados para testes
    • 

  • Sem aplicar técnica de balanceamento
    • 

  • Usando otimização de hyperparametros com Optuna com PruningCallback
    • 
















In [17]:



    


def objective(trial):
    # Vamos armazenar 'Contratar' no rótulo (y) e o restante das colunas em X
    X = dados.drop(['Contratar'], axis = 1)
    y = dados['Contratar']

    # Aplicando a mesma escala nos dados
    X = MinMaxScaler().fit_transform(X)

    # Separação dos dados em um conjunto de treino e um conjunto de teste
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
    
    dtrain = xgb.DMatrix(X_train, label=y_train)
    dvalid = xgb.DMatrix(X_test, label=y_test)

    param = {
        "silent": 1,
        "objective": "binary:logistic",
        "eval_metric": "auc",
        "booster": trial.suggest_categorical("booster", ["gbtree", "gblinear", "dart"]),
        "lambda": trial.suggest_float("lambda", 1e-8, 1.0, log=True),
        "alpha": trial.suggest_float("alpha", 1e-8, 1.0, log=True),
    }

    if param["booster"] == "gbtree" or param["booster"] == "dart":
        param["max_depth"] = trial.suggest_int("max_depth", 1, 9)
        param["eta"] = trial.suggest_float("eta", 1e-8, 1.0, log=True)
        param["gamma"] = trial.suggest_float("gamma", 1e-8, 1.0, log=True)
        param["grow_policy"] = trial.suggest_categorical("grow_policy", ["depthwise", "lossguide"])
    if param["booster"] == "dart":
        param["sample_type"] = trial.suggest_categorical("sample_type", ["uniform", "weighted"])
        param["normalize_type"] = trial.suggest_categorical("normalize_type", ["tree", "forest"])
        param["rate_drop"] = trial.suggest_float("rate_drop", 1e-8, 1.0, log=True)
        param["skip_drop"] = trial.suggest_float("skip_drop", 1e-8, 1.0, log=True)

    # Add a callback for pruning.
    pruning_callback = optuna.integration.XGBoostPruningCallback(trial, "validation-auc")
    bst = xgb.train(param, dtrain, evals=[(dvalid, "validation")], callbacks=[pruning_callback])
    preds = bst.predict(dvalid)
    pred_labels = np.rint(preds)
    accuracy = accuracy_score(y_test, pred_labels)
    return accuracy



    













In [18]:



    


study_v2 = optuna.create_study(pruner=optuna.pruners.MedianPruner(n_warmup_steps=15), direction="maximize")
study_v2.optimize(objective, n_trials=100)
print(study_v2.best_trial)



    















    






[0]	validation-auc:0.82567
[1]	validation-auc:0.82684
[2]	validation-auc:0.82757
[3]	validation-auc:0.82794
[4]	validation-auc:0.82794
[5]	validation-auc:0.82794
[6]	validation-auc:0.82794
[7]	validation-auc:0.82794
[8]	validation-auc:0.82800
[9]	validation-auc:0.82794










    






[I 2020-08-24 17:08:37,242] Finished trial#0 with value: 0.8905109489051095 with parameters: {'booster': 'dart', 'lambda': 3.3839294286710406e-07, 'alpha': 2.1846779588841234e-08, 'max_depth': 4, 'eta': 8.937529861483764e-05, 'gamma': 1.0900830202736911e-06, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'tree', 'rate_drop': 0.017899487494026606, 'skip_drop': 0.034819141888340725}. Best is trial#0 with value: 0.8905109489051095.










    






[0]	validation-auc:0.80178
[1]	validation-auc:0.79920
[2]	validation-auc:0.79743
[3]	validation-auc:0.79877
[4]	validation-auc:0.79810
[5]	validation-auc:0.78793
[6]	validation-auc:0.79179
[7]	validation-auc:0.79326
[8]	validation-auc:0.79344
[9]	validation-auc:0.79283










    






[I 2020-08-24 17:08:37,648] Finished trial#1 with value: 0.864963503649635 with parameters: {'booster': 'dart', 'lambda': 5.080767907197901e-06, 'alpha': 0.22064197179382064, 'max_depth': 9, 'eta': 5.420472129355535e-06, 'gamma': 0.0014444127245140175, 'grow_policy': 'lossguide', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 2.4610126834641843e-07, 'skip_drop': 0.0002814082261484744}. Best is trial#0 with value: 0.8905109489051095.










    






[0]	validation-auc:0.76759
[1]	validation-auc:0.77132
[2]	validation-auc:0.74700
[3]	validation-auc:0.76869
[4]	validation-auc:0.74804
[5]	validation-auc:0.77917
[6]	validation-auc:0.77996
[7]	validation-auc:0.78082
[8]	validation-auc:0.75656
[9]	validation-auc:0.78174










    






[I 2020-08-24 17:08:38,107] Finished trial#2 with value: 0.8832116788321168 with parameters: {'booster': 'gbtree', 'lambda': 0.02105204239431458, 'alpha': 0.02231890854293799, 'max_depth': 9, 'eta': 2.7924759895406137e-07, 'gamma': 3.618467795102897e-05, 'grow_policy': 'lossguide'}. Best is trial#0 with value: 0.8905109489051095.










    






[0]	validation-auc:0.51029
[1]	validation-auc:0.71495
[2]	validation-auc:0.77512
[3]	validation-auc:0.77488
[4]	validation-auc:0.76863
[5]	validation-auc:0.76348
[6]	validation-auc:0.75919
[7]	validation-auc:0.75576
[8]	validation-auc:0.75233
[9]	validation-auc:0.75074










    






[I 2020-08-24 17:08:38,327] Finished trial#3 with value: 0.8759124087591241 with parameters: {'booster': 'gblinear', 'lambda': 0.0020313845779195597, 'alpha': 0.011407092840071674}. Best is trial#0 with value: 0.8905109489051095.










    






[0]	validation-auc:0.75576
[1]	validation-auc:0.75576
[2]	validation-auc:0.75576
[3]	validation-auc:0.75576
[4]	validation-auc:0.75576
[5]	validation-auc:0.75576
[6]	validation-auc:0.75576
[7]	validation-auc:0.75576
[8]	validation-auc:0.75576
[9]	validation-auc:0.75576










    






[I 2020-08-24 17:08:38,560] Finished trial#4 with value: 0.8795620437956204 with parameters: {'booster': 'dart', 'lambda': 0.46930816047387963, 'alpha': 0.0025596210653257294, 'max_depth': 2, 'eta': 6.060245927623276e-06, 'gamma': 0.7377457890569533, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'tree', 'rate_drop': 0.00013619987337495047, 'skip_drop': 0.05429693756024944}. Best is trial#0 with value: 0.8905109489051095.










    






[0]	validation-auc:0.74828
[1]	validation-auc:0.72788
[2]	validation-auc:0.78738
[3]	validation-auc:0.75245
[4]	validation-auc:0.78039
[5]	validation-auc:0.79694
[6]	validation-auc:0.81287
[7]	validation-auc:0.82573
[8]	validation-auc:0.82353
[9]	validation-auc:0.81777










    






[I 2020-08-24 17:08:38,821] Finished trial#5 with value: 0.9014598540145985 with parameters: {'booster': 'dart', 'lambda': 2.1896914569578383e-05, 'alpha': 0.018478458596780552, 'max_depth': 5, 'eta': 0.5512296461095924, 'gamma': 0.010178213674292245, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 3.074594570782064e-05, 'skip_drop': 0.008687145192267823}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.61789
[1]	validation-auc:0.61789
[2]	validation-auc:0.61789
[3]	validation-auc:0.61789
[4]	validation-auc:0.61789
[5]	validation-auc:0.61789
[6]	validation-auc:0.61789
[7]	validation-auc:0.61789
[8]	validation-auc:0.61789
[9]	validation-auc:0.61789










    






[I 2020-08-24 17:08:39,029] Finished trial#6 with value: 0.8759124087591241 with parameters: {'booster': 'gbtree', 'lambda': 0.049905562919399134, 'alpha': 0.5874859622123816, 'max_depth': 1, 'eta': 2.0982749166714264e-06, 'gamma': 0.0005162545569444887, 'grow_policy': 'lossguide'}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.78952
[1]	validation-auc:0.79761
[2]	validation-auc:0.75355
[3]	validation-auc:0.76140
[4]	validation-auc:0.80098
[5]	validation-auc:0.80748
[6]	validation-auc:0.80882
[7]	validation-auc:0.81985
[8]	validation-auc:0.83223
[9]	validation-auc:0.82757










    






[I 2020-08-24 17:08:39,376] Finished trial#7 with value: 0.9014598540145985 with parameters: {'booster': 'dart', 'lambda': 0.00039398131602786626, 'alpha': 1.0109600297379858e-08, 'max_depth': 9, 'eta': 0.8525258561558186, 'gamma': 1.812464971888174e-08, 'grow_policy': 'lossguide', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 0.0281217307839915, 'skip_drop': 0.0010987883230068084}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.74608
[1]	validation-auc:0.77169
[2]	validation-auc:0.75974
[3]	validation-auc:0.77316
[4]	validation-auc:0.77328
[5]	validation-auc:0.75993
[6]	validation-auc:0.77310
[7]	validation-auc:0.77328
[8]	validation-auc:0.77322
[9]	validation-auc:0.77328










    






[I 2020-08-24 17:08:39,731] Finished trial#8 with value: 0.9014598540145985 with parameters: {'booster': 'gbtree', 'lambda': 0.0030705627399960807, 'alpha': 6.35311852950201e-08, 'max_depth': 5, 'eta': 0.00030884662414580366, 'gamma': 0.0038205436877559577, 'grow_policy': 'lossguide'}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.77469
[1]	validation-auc:0.77469
[2]	validation-auc:0.77469
[3]	validation-auc:0.77469
[4]	validation-auc:0.77469
[5]	validation-auc:0.77469
[6]	validation-auc:0.77469
[7]	validation-auc:0.77469
[8]	validation-auc:0.77469
[9]	validation-auc:0.77469










    






[I 2020-08-24 17:08:40,118] Finished trial#9 with value: 0.8978102189781022 with parameters: {'booster': 'dart', 'lambda': 9.192301530009855e-06, 'alpha': 0.0005145627736325201, 'max_depth': 3, 'eta': 7.089810060737344e-06, 'gamma': 0.032621228837253685, 'grow_policy': 'lossguide', 'sample_type': 'weighted', 'normalize_type': 'tree', 'rate_drop': 0.4242664374931617, 'skip_drop': 4.8464245399765974e-05}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.44522
[1]	validation-auc:0.72071
[2]	validation-auc:0.78100
[3]	validation-auc:0.79326
[4]	validation-auc:0.80931
[5]	validation-auc:0.81446
[6]	validation-auc:0.82047
[7]	validation-auc:0.82108
[8]	validation-auc:0.82279
[9]	validation-auc:0.82218










    






[I 2020-08-24 17:08:40,461] Finished trial#10 with value: 0.8832116788321168 with parameters: {'booster': 'gblinear', 'lambda': 1.3194630298269746e-08, 'alpha': 6.626718239783707e-06}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.74571
[1]	validation-auc:0.76985
[2]	validation-auc:0.77549
[3]	validation-auc:0.77145
[4]	validation-auc:0.77138
[5]	validation-auc:0.77151
[6]	validation-auc:0.77151
[7]	validation-auc:0.77151
[8]	validation-auc:0.76955
[9]	validation-auc:0.77151










    






[I 2020-08-24 17:08:40,830] Finished trial#11 with value: 0.9014598540145985 with parameters: {'booster': 'dart', 'lambda': 9.989604844638997e-05, 'alpha': 1.1089256726955688e-05, 'max_depth': 5, 'eta': 0.00016417836418733427, 'gamma': 1.0106666770079123e-05, 'grow_policy': 'lossguide', 'sample_type': 'uniform', 'normalize_type': 'tree', 'rate_drop': 8.627880815227218e-08, 'skip_drop': 8.184601075309658e-07}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.69945
[1]	validation-auc:0.78413
[2]	validation-auc:0.77770
[3]	validation-auc:0.78536
[4]	validation-auc:0.78033
[5]	validation-auc:0.78517
[6]	validation-auc:0.78382
[7]	validation-auc:0.78493
[8]	validation-auc:0.79534
[9]	validation-auc:0.79620










    






[I 2020-08-24 17:08:41,202] Finished trial#12 with value: 0.8941605839416058 with parameters: {'booster': 'dart', 'lambda': 0.00011668268726602602, 'alpha': 2.2443345703678513e-06, 'max_depth': 7, 'eta': 0.5349590149873886, 'gamma': 0.3088805816461933, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 9.266078433659649e-08, 'skip_drop': 0.005808179274962702}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.69945
[1]	validation-auc:0.72653
[2]	validation-auc:0.74982
[3]	validation-auc:0.82616
[4]	validation-auc:0.82267
[5]	validation-auc:0.84510
[6]	validation-auc:0.85380
[7]	validation-auc:0.86765
[8]	validation-auc:0.85184
[9]	validation-auc:0.84265










    






[I 2020-08-24 17:08:41,682] Finished trial#13 with value: 0.9014598540145985 with parameters: {'booster': 'dart', 'lambda': 0.00011747173675712783, 'alpha': 0.0002011697849474967, 'max_depth': 7, 'eta': 0.36843887293172783, 'gamma': 1.1219039621575428e-08, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 0.019685832813566735, 'skip_drop': 0.0018959976545273448}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.67788
[1]	validation-auc:0.75123
[2]	validation-auc:0.73315
[3]	validation-auc:0.74461
[4]	validation-auc:0.73897
[5]	validation-auc:0.73505
[6]	validation-auc:0.73719
[7]	validation-auc:0.76397
[8]	validation-auc:0.77782
[9]	validation-auc:0.78505










    






[I 2020-08-24 17:08:42,083] Finished trial#14 with value: 0.8905109489051095 with parameters: {'booster': 'dart', 'lambda': 7.472108741652496e-07, 'alpha': 2.0766485158023998e-07, 'max_depth': 7, 'eta': 0.02003737023179418, 'gamma': 1.675440558113498e-08, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'forest', 'rate_drop': 0.0008285191124585368, 'skip_drop': 2.3235486538442575e-05}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.57764
[1]	validation-auc:0.57764
[2]	validation-auc:0.57764
[3]	validation-auc:0.57764
[4]	validation-auc:0.57764
[5]	validation-auc:0.57764
[6]	validation-auc:0.57764
[7]	validation-auc:0.57764
[8]	validation-auc:0.57751
[9]	validation-auc:0.57764










    






[I 2020-08-24 17:08:42,306] Finished trial#15 with value: 0.8759124087591241 with parameters: {'booster': 'gblinear', 'lambda': 0.0007273983616938733, 'alpha': 0.10806763627820068}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.76556
[1]	validation-auc:0.77359
[2]	validation-auc:0.75570
[3]	validation-auc:0.75999
[4]	validation-auc:0.75276
[5]	validation-auc:0.74847
[6]	validation-auc:0.74455
[7]	validation-auc:0.76042
[8]	validation-auc:0.75637
[9]	validation-auc:0.77132










    






[I 2020-08-24 17:08:42,679] Finished trial#16 with value: 0.8905109489051095 with parameters: {'booster': 'dart', 'lambda': 8.6140735929288e-06, 'alpha': 4.770350097796105e-05, 'max_depth': 8, 'eta': 0.020813639126821472, 'gamma': 4.2613386314617555e-07, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 5.767471644826193e-06, 'skip_drop': 0.9957515230508714}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.74816
[1]	validation-auc:0.79559
[2]	validation-auc:0.80649
[3]	validation-auc:0.80343
[4]	validation-auc:0.81765
[5]	validation-auc:0.82157
[6]	validation-auc:0.82059
[7]	validation-auc:0.83217
[8]	validation-auc:0.83094
[9]	validation-auc:0.82653










    






[I 2020-08-24 17:08:43,073] Finished trial#17 with value: 0.8978102189781022 with parameters: {'booster': 'gbtree', 'lambda': 0.00652911536762701, 'alpha': 4.4904106859241586e-07, 'max_depth': 6, 'eta': 0.029698830510821138, 'gamma': 0.016691990258159522, 'grow_policy': 'depthwise'}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.77469
[1]	validation-auc:0.78107
[2]	validation-auc:0.76029
[3]	validation-auc:0.80846
[4]	validation-auc:0.80024
[5]	validation-auc:0.78811
[6]	validation-auc:0.80564
[7]	validation-auc:0.80325
[8]	validation-auc:0.81115
[9]	validation-auc:0.80980










    






[I 2020-08-24 17:08:43,434] Finished trial#18 with value: 0.9014598540145985 with parameters: {'booster': 'gbtree', 'lambda': 0.6724817103459327, 'alpha': 0.00022876937047659613, 'max_depth': 3, 'eta': 0.7980295221245152, 'gamma': 0.12496695508057325, 'grow_policy': 'lossguide'}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.50000
[1]	validation-auc:0.50000
[2]	validation-auc:0.50000
[3]	validation-auc:0.50000
[4]	validation-auc:0.50000
[5]	validation-auc:0.50000
[6]	validation-auc:0.50000
[7]	validation-auc:0.61789
[8]	validation-auc:0.61789
[9]	validation-auc:0.61789










    






[I 2020-08-24 17:08:43,859] Finished trial#19 with value: 0.8759124087591241 with parameters: {'booster': 'gbtree', 'lambda': 0.9530223037193729, 'alpha': 0.0002996831966530686, 'max_depth': 1, 'eta': 1.609159165104825e-08, 'gamma': 0.0002418542239620658, 'grow_policy': 'depthwise'}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.74816
[1]	validation-auc:0.76759
[2]	validation-auc:0.76042
[3]	validation-auc:0.76887
[4]	validation-auc:0.76495
[5]	validation-auc:0.76415
[6]	validation-auc:0.76808
[7]	validation-auc:0.76734
[8]	validation-auc:0.76881
[9]	validation-auc:0.77022










    






[I 2020-08-24 17:08:44,142] Finished trial#20 with value: 0.9014598540145985 with parameters: {'booster': 'dart', 'lambda': 1.0571154187241398e-07, 'alpha': 0.0037558486660038645, 'max_depth': 6, 'eta': 0.00285809809439284, 'gamma': 2.3371957475347114e-07, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 5.990250093131202e-06, 'skip_drop': 0.8834420239278066}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.77469
[1]	validation-auc:0.77469
[2]	validation-auc:0.77469
[3]	validation-auc:0.77469
[4]	validation-auc:0.77469
[5]	validation-auc:0.77469
[6]	validation-auc:0.77469
[7]	validation-auc:0.77469
[8]	validation-auc:0.77469
[9]	validation-auc:0.77469










    






[I 2020-08-24 17:08:44,522] Finished trial#21 with value: 0.8978102189781022 with parameters: {'booster': 'gbtree', 'lambda': 1.9209371190566722e-08, 'alpha': 0.0017195226229461957, 'max_depth': 3, 'eta': 0.0017207690748539772, 'gamma': 1.6554853087142364e-07, 'grow_policy': 'lossguide'}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.50000
[1]	validation-auc:0.50000
[2]	validation-auc:0.50000
[3]	validation-auc:0.55674
[4]	validation-auc:0.58615
[5]	validation-auc:0.57145
[6]	validation-auc:0.58615
[7]	validation-auc:0.73952
[8]	validation-auc:0.77886
[9]	validation-auc:0.77586










    






[I 2020-08-24 17:08:45,046] Finished trial#22 with value: 0.8941605839416058 with parameters: {'booster': 'gbtree', 'lambda': 2.4645025458951437e-05, 'alpha': 5.994787439647212e-05, 'max_depth': 4, 'eta': 1.2521320304809519e-08, 'gamma': 1.199587701663709e-05, 'grow_policy': 'lossguide'}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.74510
[1]	validation-auc:0.74510
[2]	validation-auc:0.74510
[3]	validation-auc:0.74510
[4]	validation-auc:0.74779
[5]	validation-auc:0.75349
[6]	validation-auc:0.75374
[7]	validation-auc:0.75447
[8]	validation-auc:0.75264
[9]	validation-auc:0.75717










    






[I 2020-08-24 17:08:45,289] Finished trial#23 with value: 0.8868613138686131 with parameters: {'booster': 'dart', 'lambda': 1.9119372028057665e-07, 'alpha': 2.2458892933927897e-05, 'max_depth': 6, 'eta': 0.001060670990155447, 'gamma': 3.869529215033917e-06, 'grow_policy': 'lossguide', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 0.9259830368903503, 'skip_drop': 2.0290123072365384e-06}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.48039
[1]	validation-auc:0.50931
[2]	validation-auc:0.60601
[3]	validation-auc:0.68860
[4]	validation-auc:0.72194
[5]	validation-auc:0.74240
[6]	validation-auc:0.75956
[7]	validation-auc:0.77390
[8]	validation-auc:0.78100
[9]	validation-auc:0.79314










    






[I 2020-08-24 17:08:45,567] Finished trial#24 with value: 0.8832116788321168 with parameters: {'booster': 'gblinear', 'lambda': 5.911527979185857e-08, 'alpha': 6.031534955262395e-06}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.78903
[1]	validation-auc:0.81526
[2]	validation-auc:0.80282
[3]	validation-auc:0.80876
[4]	validation-auc:0.80974
[5]	validation-auc:0.80974
[6]	validation-auc:0.82175
[7]	validation-auc:0.81060
[8]	validation-auc:0.82230
[9]	validation-auc:0.83052










    






[I 2020-08-24 17:08:45,842] Finished trial#25 with value: 0.9014598540145985 with parameters: {'booster': 'dart', 'lambda': 0.1549382102859049, 'alpha': 0.017426685923020185, 'max_depth': 4, 'eta': 0.13442746196788605, 'gamma': 0.09926759672464543, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 0.002993195899957012, 'skip_drop': 0.0008547452225154555}. Best is trial#5 with value: 0.9014598540145985.










    






[0]	validation-auc:0.78903
[1]	validation-auc:0.80135
[2]	validation-auc:0.79246
[3]	validation-auc:0.79289
[4]	validation-auc:0.80502
[5]	validation-auc:0.80637
[6]	validation-auc:0.81440
[7]	validation-auc:0.81538
[8]	validation-auc:0.81777
[9]	validation-auc:0.80772










    






[I 2020-08-24 17:08:46,097] Finished trial#26 with value: 0.9051094890510949 with parameters: {'booster': 'gbtree', 'lambda': 0.19353951281099527, 'alpha': 0.002923480246036488, 'max_depth': 4, 'eta': 0.11243192241078481, 'gamma': 0.011934614571296562, 'grow_policy': 'depthwise'}. Best is trial#26 with value: 0.9051094890510949.










    






[0]	validation-auc:0.77469
[1]	validation-auc:0.77469
[2]	validation-auc:0.77469
[3]	validation-auc:0.77469
[4]	validation-auc:0.77469
[5]	validation-auc:0.77469
[6]	validation-auc:0.77469
[7]	validation-auc:0.77469
[8]	validation-auc:0.77469
[9]	validation-auc:0.77469










    






[I 2020-08-24 17:08:46,434] Finished trial#27 with value: 0.8978102189781022 with parameters: {'booster': 'gbtree', 'lambda': 0.18953908446700224, 'alpha': 0.0015427484293014891, 'max_depth': 3, 'eta': 0.0034102919029269847, 'gamma': 0.07961933936100905, 'grow_policy': 'depthwise'}. Best is trial#26 with value: 0.9051094890510949.










    






[0]	validation-auc:0.78903
[1]	validation-auc:0.80135
[2]	validation-auc:0.79246
[3]	validation-auc:0.81661
[4]	validation-auc:0.82194
[5]	validation-auc:0.81826
[6]	validation-auc:0.81317
[7]	validation-auc:0.81759
[8]	validation-auc:0.81109
[9]	validation-auc:0.81336










    






[I 2020-08-24 17:08:46,896] Finished trial#28 with value: 0.9051094890510949 with parameters: {'booster': 'dart', 'lambda': 0.06630755691193725, 'alpha': 0.05327209946587316, 'max_depth': 4, 'eta': 0.10091272314913156, 'gamma': 0.5315800506734761, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 6.740436695550875e-06, 'skip_drop': 2.7808441481925832e-08}. Best is trial#26 with value: 0.9051094890510949.










    






[0]	validation-auc:0.78903
[1]	validation-auc:0.80129
[2]	validation-auc:0.79332
[3]	validation-auc:0.81605
[4]	validation-auc:0.78934
[5]	validation-auc:0.79669
[6]	validation-auc:0.80049
[7]	validation-auc:0.80864
[8]	validation-auc:0.81716
[9]	validation-auc:0.82378










    






[I 2020-08-24 17:08:47,155] Finished trial#29 with value: 0.9124087591240876 with parameters: {'booster': 'dart', 'lambda': 0.09677610590815423, 'alpha': 0.112577980226669, 'max_depth': 4, 'eta': 0.09562096050740938, 'gamma': 0.6379847053082929, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 3.7196935870056373e-06, 'skip_drop': 1.5021697707098247e-08}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.75576
[1]	validation-auc:0.75576
[2]	validation-auc:0.75576
[3]	validation-auc:0.76863
[4]	validation-auc:0.77096
[5]	validation-auc:0.75699
[6]	validation-auc:0.76887
[7]	validation-auc:0.74890
[8]	validation-auc:0.76514
[9]	validation-auc:0.77138










    






[I 2020-08-24 17:08:47,408] Finished trial#30 with value: 0.8795620437956204 with parameters: {'booster': 'dart', 'lambda': 0.025268546104179015, 'alpha': 0.1024069754738074, 'max_depth': 2, 'eta': 0.08479427422093862, 'gamma': 0.8624741053245479, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 1.5805150203112122e-06, 'skip_drop': 1.4123255372488777e-08}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.79142
[1]	validation-auc:0.79228
[2]	validation-auc:0.79424
[3]	validation-auc:0.77181
[4]	validation-auc:0.77206
[5]	validation-auc:0.77567
[6]	validation-auc:0.78352
[7]	validation-auc:0.79773
[8]	validation-auc:0.79939
[9]	validation-auc:0.80460










    






[I 2020-08-24 17:08:47,668] Finished trial#31 with value: 0.9087591240875912 with parameters: {'booster': 'dart', 'lambda': 0.08343221379490538, 'alpha': 0.7422293455612119, 'max_depth': 4, 'eta': 0.09273131430775083, 'gamma': 0.9312749572604749, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'forest', 'rate_drop': 1.2542860191308608e-08, 'skip_drop': 1.798526348873933e-08}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.75576
[1]	validation-auc:0.75576
[2]	validation-auc:0.75576
[3]	validation-auc:0.75576
[4]	validation-auc:0.75576
[5]	validation-auc:0.75576
[6]	validation-auc:0.75576
[7]	validation-auc:0.75576
[8]	validation-auc:0.75576
[9]	validation-auc:0.75576










    






[I 2020-08-24 17:08:47,920] Finished trial#32 with value: 0.8795620437956204 with parameters: {'booster': 'dart', 'lambda': 0.09172232967406475, 'alpha': 0.7297677700654124, 'max_depth': 2, 'eta': 0.0096807435376878, 'gamma': 0.9320911101076585, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'forest', 'rate_drop': 6.733580433060647e-07, 'skip_drop': 1.1162557620895535e-07}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.78903
[1]	validation-auc:0.79026
[2]	validation-auc:0.79099
[3]	validation-auc:0.79136
[4]	validation-auc:0.79063
[5]	validation-auc:0.79136
[6]	validation-auc:0.79105
[7]	validation-auc:0.79142
[8]	validation-auc:0.79142
[9]	validation-auc:0.79142










    






[I 2020-08-24 17:08:48,305] Finished trial#33 with value: 0.8941605839416058 with parameters: {'booster': 'dart', 'lambda': 0.010540167780513948, 'alpha': 0.09099729982465124, 'max_depth': 4, 'eta': 4.67254871275356e-05, 'gamma': 0.34639218181711584, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'forest', 'rate_drop': 1.7230273966824506e-08, 'skip_drop': 1.9162835443984003e-07}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.78051
[1]	validation-auc:0.76771
[2]	validation-auc:0.78873
[3]	validation-auc:0.79222
[4]	validation-auc:0.80692
[5]	validation-auc:0.81373
[6]	validation-auc:0.81507
[7]	validation-auc:0.82996
[8]	validation-auc:0.83805
[9]	validation-auc:0.84589










    






[I 2020-08-24 17:08:48,698] Finished trial#34 with value: 0.8978102189781022 with parameters: {'booster': 'dart', 'lambda': 0.3185051343632201, 'alpha': 0.37162755319906027, 'max_depth': 5, 'eta': 0.09204934542775489, 'gamma': 0.23954897009139428, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'forest', 'rate_drop': 1.0699559582762633e-08, 'skip_drop': 5.50302310694805e-06}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.78903
[1]	validation-auc:0.80312
[2]	validation-auc:0.80343
[3]	validation-auc:0.80625
[4]	validation-auc:0.80717
[5]	validation-auc:0.81912
[6]	validation-auc:0.83137
[7]	validation-auc:0.82341
[8]	validation-auc:0.82598
[9]	validation-auc:0.82102










    






[I 2020-08-24 17:08:49,081] Finished trial#35 with value: 0.8978102189781022 with parameters: {'booster': 'dart', 'lambda': 0.028853840794955517, 'alpha': 0.06189646305862133, 'max_depth': 4, 'eta': 0.19804561368838758, 'gamma': 0.04313453608633497, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'forest', 'rate_drop': 5.0005655146781796e-05, 'skip_drop': 1.1007202747630275e-08}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.78903
[1]	validation-auc:0.81691
[2]	validation-auc:0.78168
[3]	validation-auc:0.81262
[4]	validation-auc:0.82090
[5]	validation-auc:0.81814
[6]	validation-auc:0.79669
[7]	validation-auc:0.80772
[8]	validation-auc:0.80809
[9]	validation-auc:0.81250










    






[I 2020-08-24 17:08:49,350] Finished trial#36 with value: 0.9014598540145985 with parameters: {'booster': 'dart', 'lambda': 0.07264895592858783, 'alpha': 0.00653416026443959, 'max_depth': 4, 'eta': 0.05352344809916217, 'gamma': 0.004581197527131561, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'forest', 'rate_drop': 5.061629122632495e-06, 'skip_drop': 5.97388955033334e-08}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.60025
[1]	validation-auc:0.62414
[2]	validation-auc:0.62598
[3]	validation-auc:0.58983
[4]	validation-auc:0.56299
[5]	validation-auc:0.54387
[6]	validation-auc:0.54387
[7]	validation-auc:0.54393
[8]	validation-auc:0.54387
[9]	validation-auc:0.54393










    






[I 2020-08-24 17:08:49,564] Finished trial#37 with value: 0.8759124087591241 with parameters: {'booster': 'gblinear', 'lambda': 0.0085261791728252, 'alpha': 0.03787088179483307}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.77469
[1]	validation-auc:0.77469
[2]	validation-auc:0.77469
[3]	validation-auc:0.77469
[4]	validation-auc:0.77469
[5]	validation-auc:0.77469
[6]	validation-auc:0.77469
[7]	validation-auc:0.77469
[8]	validation-auc:0.77469
[9]	validation-auc:0.77469










    






[I 2020-08-24 17:08:49,792] Finished trial#38 with value: 0.8978102189781022 with parameters: {'booster': 'gbtree', 'lambda': 0.9865116645434698, 'alpha': 0.9916218503743404, 'max_depth': 3, 'eta': 0.00701679150055775, 'gamma': 0.7733478385735519, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.79075
[1]	validation-auc:0.80239
[2]	validation-auc:0.80355
[3]	validation-auc:0.82923
[4]	validation-auc:0.82353
[5]	validation-auc:0.81415
[6]	validation-auc:0.82923
[7]	validation-auc:0.81685
[8]	validation-auc:0.81556
[9]	validation-auc:0.80698










    






[I 2020-08-24 17:08:50,201] Finished trial#39 with value: 0.8941605839416058 with parameters: {'booster': 'dart', 'lambda': 0.0019998948999736563, 'alpha': 0.2219014079219198, 'max_depth': 4, 'eta': 0.20617161502872516, 'gamma': 0.0027443387467573394, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 0.00029870774397628173, 'skip_drop': 1.2823763608974649e-08}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.75233
[1]	validation-auc:0.75938
[2]	validation-auc:0.77120
[3]	validation-auc:0.76857
[4]	validation-auc:0.78113
[5]	validation-auc:0.80931
[6]	validation-auc:0.80478
[7]	validation-auc:0.81961
[8]	validation-auc:0.81556
[9]	validation-auc:0.82243










    






[I 2020-08-24 17:08:50,630] Finished trial#40 with value: 0.9051094890510949 with parameters: {'booster': 'dart', 'lambda': 0.3075715184926798, 'alpha': 0.02579069063409967, 'max_depth': 5, 'eta': 0.044083060120229875, 'gamma': 0.017442972974623192, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'forest', 'rate_drop': 1.5880811999791225e-05, 'skip_drop': 4.426019356899594e-07}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.75233
[1]	validation-auc:0.75938
[2]	validation-auc:0.75901
[3]	validation-auc:0.75987
[4]	validation-auc:0.75938
[5]	validation-auc:0.75962
[6]	validation-auc:0.75974
[7]	validation-auc:0.75962
[8]	validation-auc:0.75974
[9]	validation-auc:0.75987










    






[I 2020-08-24 17:08:51,063] Finished trial#41 with value: 0.9014598540145985 with parameters: {'booster': 'dart', 'lambda': 0.26218704143777244, 'alpha': 0.008605794864052553, 'max_depth': 5, 'eta': 0.007135857026518367, 'gamma': 0.0007008874317035653, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 1.0645669507080403e-06, 'skip_drop': 4.057575866379476e-08}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.73241
[1]	validation-auc:0.75392
[2]	validation-auc:0.74877
[3]	validation-auc:0.75552
[4]	validation-auc:0.75552
[5]	validation-auc:0.74975
[6]	validation-auc:0.75588
[7]	validation-auc:0.74988
[8]	validation-auc:0.75576
[9]	validation-auc:0.75588










    






[I 2020-08-24 17:08:51,414] Finished trial#42 with value: 0.8868613138686131 with parameters: {'booster': 'dart', 'lambda': 0.05114487486988568, 'alpha': 0.03810490311898312, 'max_depth': 6, 'eta': 0.0005940105495176279, 'gamma': 7.474650518045799e-05, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 5.6817928597864e-08, 'skip_drop': 3.044510635256873e-08}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.75576
[1]	validation-auc:0.75576
[2]	validation-auc:0.75576
[3]	validation-auc:0.75576
[4]	validation-auc:0.75576
[5]	validation-auc:0.74902
[6]	validation-auc:0.74902
[7]	validation-auc:0.75846
[8]	validation-auc:0.76471
[9]	validation-auc:0.76948










    






[I 2020-08-24 17:08:51,645] Finished trial#43 with value: 0.8795620437956204 with parameters: {'booster': 'dart', 'lambda': 0.1295816237375003, 'alpha': 0.2694901439615812, 'max_depth': 2, 'eta': 0.04498235188193119, 'gamma': 0.015003266742039166, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'tree', 'rate_drop': 1.760030347334267e-05, 'skip_drop': 4.3110393642439593e-07}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.78051
[1]	validation-auc:0.71091
[2]	validation-auc:0.77911
[3]	validation-auc:0.77010
[4]	validation-auc:0.78033
[5]	validation-auc:0.79087
[6]	validation-auc:0.79148
[7]	validation-auc:0.76605
[8]	validation-auc:0.76434
[9]	validation-auc:0.74393










    






[I 2020-08-24 17:08:51,901] Finished trial#44 with value: 0.8978102189781022 with parameters: {'booster': 'dart', 'lambda': 0.4129011736458795, 'alpha': 0.02318348609301124, 'max_depth': 5, 'eta': 0.36606316245879156, 'gamma': 0.0010546820015789001, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'forest', 'rate_drop': 0.0005854405745363835, 'skip_drop': 4.577206964478863e-06}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.77469
[1]	validation-auc:0.77469
[2]	validation-auc:0.77469
[3]	validation-auc:0.76250
[4]	validation-auc:0.76250
[5]	validation-auc:0.76348
[6]	validation-auc:0.76348
[7]	validation-auc:0.76348
[8]	validation-auc:0.76348
[9]	validation-auc:0.76348










    






[I 2020-08-24 17:08:52,168] Finished trial#45 with value: 0.8941605839416058 with parameters: {'booster': 'dart', 'lambda': 0.014670315604172323, 'alpha': 0.004107069468659916, 'max_depth': 3, 'eta': 0.013501247133172108, 'gamma': 0.3411228449634121, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'tree', 'rate_drop': 3.25819698083763e-07, 'skip_drop': 1.421380803106139e-07}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.82555
[1]	validation-auc:0.77347
[2]	validation-auc:0.81195
[3]	validation-auc:0.81906
[4]	validation-auc:0.82028
[5]	validation-auc:0.80729
[6]	validation-auc:0.80288
[7]	validation-auc:0.80245
[8]	validation-auc:0.81262
[9]	validation-auc:0.81544










    






[I 2020-08-24 17:08:52,447] Finished trial#46 with value: 0.8978102189781022 with parameters: {'booster': 'dart', 'lambda': 0.003511026333583747, 'alpha': 0.001071650373219992, 'max_depth': 4, 'eta': 0.06700206168463471, 'gamma': 0.04100008483365356, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'forest', 'rate_drop': 3.125520127095491e-08, 'skip_drop': 1.5241812548058995e-08}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.79320
[1]	validation-auc:0.82310
[2]	validation-auc:0.79099
[3]	validation-auc:0.79283
[4]	validation-auc:0.79001
[5]	validation-auc:0.79975
[6]	validation-auc:0.81471
[7]	validation-auc:0.82727
[8]	validation-auc:0.81219
[9]	validation-auc:0.82837










    






[I 2020-08-24 17:08:52,737] Finished trial#47 with value: 0.9014598540145985 with parameters: {'booster': 'gbtree', 'lambda': 0.025889551772256842, 'alpha': 0.9862621788820485, 'max_depth': 4, 'eta': 0.21603833994053312, 'gamma': 0.006810027747391071, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.78652
[1]	validation-auc:0.73897
[2]	validation-auc:0.72414
[3]	validation-auc:0.73548
[4]	validation-auc:0.76771
[5]	validation-auc:0.76611
[6]	validation-auc:0.77714
[7]	validation-auc:0.77825
[8]	validation-auc:0.78762
[9]	validation-auc:0.79473










    






[I 2020-08-24 17:08:53,158] Finished trial#48 with value: 0.9051094890510949 with parameters: {'booster': 'dart', 'lambda': 0.5223532482538472, 'alpha': 0.17321659579408516, 'max_depth': 5, 'eta': 0.8255993917386807, 'gamma': 0.1980199861746613, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'forest', 'rate_drop': 1.2458255211397793e-05, 'skip_drop': 5.274390248328433e-07}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.56789
[1]	validation-auc:0.70147
[2]	validation-auc:0.72181
[3]	validation-auc:0.73677
[4]	validation-auc:0.73248
[5]	validation-auc:0.72990
[6]	validation-auc:0.72745
[7]	validation-auc:0.72476
[8]	validation-auc:0.72181
[9]	validation-auc:0.72414










    






[I 2020-08-24 17:08:53,391] Finished trial#49 with value: 0.8759124087591241 with parameters: {'booster': 'gblinear', 'lambda': 0.06541813306832558, 'alpha': 0.014188802582189587}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.78652
[1]	validation-auc:0.78309
[2]	validation-auc:0.75527
[3]	validation-auc:0.77071
[4]	validation-auc:0.79118
[5]	validation-auc:0.80024
[6]	validation-auc:0.81176
[7]	validation-auc:0.82304
[8]	validation-auc:0.81495
[9]	validation-auc:0.83701










    






[I 2020-08-24 17:08:53,634] Finished trial#50 with value: 0.8941605839416058 with parameters: {'booster': 'dart', 'lambda': 0.6303763547449163, 'alpha': 0.1607170739517924, 'max_depth': 5, 'eta': 0.6014740300409644, 'gamma': 0.17282978492156956, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'forest', 'rate_drop': 7.203064760926191e-05, 'skip_drop': 4.85586652886941e-08}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.77469
[1]	validation-auc:0.77022
[2]	validation-auc:0.77341
[3]	validation-auc:0.81262
[4]	validation-auc:0.82910
[5]	validation-auc:0.81085
[6]	validation-auc:0.83162
[7]	validation-auc:0.82120
[8]	validation-auc:0.79743
[9]	validation-auc:0.82071










    






[I 2020-08-24 17:08:53,919] Finished trial#51 with value: 0.8759124087591241 with parameters: {'booster': 'dart', 'lambda': 0.2901476623144627, 'alpha': 0.37310790953861, 'max_depth': 3, 'eta': 0.39764480108223915, 'gamma': 0.39443912535153564, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'forest', 'rate_drop': 3.172013142369866e-06, 'skip_drop': 1.059552430417882e-08}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.79210
[1]	validation-auc:0.79246
[2]	validation-auc:0.79259
[3]	validation-auc:0.79283
[4]	validation-auc:0.79210
[5]	validation-auc:0.79283
[6]	validation-auc:0.79289
[7]	validation-auc:0.79289
[8]	validation-auc:0.79289
[9]	validation-auc:0.79289










    






[I 2020-08-24 17:08:54,295] Finished trial#52 with value: 0.8978102189781022 with parameters: {'booster': 'dart', 'lambda': 0.9758378725632434, 'alpha': 0.054383504703277734, 'max_depth': 4, 'eta': 2.8908871682511896e-05, 'gamma': 0.8928818874439374, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'forest', 'rate_drop': 1.760591224695491e-05, 'skip_drop': 6.135799572378467e-07}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.74798
[1]	validation-auc:0.77077
[2]	validation-auc:0.77567
[3]	validation-auc:0.74614
[4]	validation-auc:0.78051
[5]	validation-auc:0.80576
[6]	validation-auc:0.82083
[7]	validation-auc:0.83431
[8]	validation-auc:0.83615
[9]	validation-auc:0.82757










    






[I 2020-08-24 17:08:54,734] Finished trial#53 with value: 0.8795620437956204 with parameters: {'booster': 'dart', 'lambda': 0.14672727769329177, 'alpha': 0.027262353591772308, 'max_depth': 6, 'eta': 0.8671674130689146, 'gamma': 0.0605750776127451, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'forest', 'rate_drop': 3.902216565161803e-07, 'skip_drop': 1.600102095414114e-07}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.78677
[1]	validation-auc:0.76514
[2]	validation-auc:0.74240
[3]	validation-auc:0.75925
[4]	validation-auc:0.75840
[5]	validation-auc:0.75619
[6]	validation-auc:0.74246
[7]	validation-auc:0.74963
[8]	validation-auc:0.74816
[9]	validation-auc:0.75245










    






[I 2020-08-24 17:08:55,029] Finished trial#54 with value: 0.8832116788321168 with parameters: {'booster': 'dart', 'lambda': 0.5221096568494467, 'alpha': 0.5553755975918516, 'max_depth': 5, 'eta': 0.903689462195447, 'gamma': 0.4967271699835373, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'forest', 'rate_drop': 0.0001684617057820543, 'skip_drop': 1.229088708496903e-05}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.75000
[1]	validation-auc:0.76048
[2]	validation-auc:0.77169
[3]	validation-auc:0.75331
[4]	validation-auc:0.75772
[5]	validation-auc:0.77175
[6]	validation-auc:0.78431
[7]	validation-auc:0.80410
[8]	validation-auc:0.81115
[9]	validation-auc:0.81422










    






[I 2020-08-24 17:08:55,361] Finished trial#55 with value: 0.9051094890510949 with parameters: {'booster': 'dart', 'lambda': 0.06028502314343388, 'alpha': 0.13373154961609024, 'max_depth': 5, 'eta': 0.03251334036115828, 'gamma': 0.026697373666258944, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'tree', 'rate_drop': 1.8689022851955275e-06, 'skip_drop': 1.5236166136665714e-06}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.75135
[1]	validation-auc:0.77739
[2]	validation-auc:0.76550
[3]	validation-auc:0.75919
[4]	validation-auc:0.76109
[5]	validation-auc:0.75270
[6]	validation-auc:0.75398
[7]	validation-auc:0.75833
[8]	validation-auc:0.75233
[9]	validation-auc:0.75172










    






[I 2020-08-24 17:08:55,629] Finished trial#56 with value: 0.8832116788321168 with parameters: {'booster': 'gbtree', 'lambda': 0.03776492193999232, 'alpha': 0.1373998900714593, 'max_depth': 6, 'eta': 4.110636616319315e-07, 'gamma': 0.15342872799175092, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.75257
[1]	validation-auc:0.76881
[2]	validation-auc:0.77194
[3]	validation-auc:0.77616
[4]	validation-auc:0.75656
[5]	validation-auc:0.76097
[6]	validation-auc:0.75337
[7]	validation-auc:0.78064
[8]	validation-auc:0.78370
[9]	validation-auc:0.80263










    






[I 2020-08-24 17:08:55,890] Finished trial#57 with value: 0.9014598540145985 with parameters: {'booster': 'dart', 'lambda': 0.004615666704669991, 'alpha': 0.06843130531628486, 'max_depth': 5, 'eta': 0.025836080942037085, 'gamma': 0.01773619480016108, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'tree', 'rate_drop': 1.49311102575704e-07, 'skip_drop': 1.7156501998372298e-06}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.75110
[1]	validation-auc:0.75889
[2]	validation-auc:0.75355
[3]	validation-auc:0.76562
[4]	validation-auc:0.77690
[5]	validation-auc:0.80294
[6]	validation-auc:0.80999
[7]	validation-auc:0.81115
[8]	validation-auc:0.82414
[9]	validation-auc:0.81942










    






[I 2020-08-24 17:08:56,137] Finished trial#58 with value: 0.9014598540145985 with parameters: {'booster': 'dart', 'lambda': 0.0007835831964466005, 'alpha': 0.4551809387584009, 'max_depth': 5, 'eta': 0.037334815143435715, 'gamma': 0.024602289903052455, 'grow_policy': 'depthwise', 'sample_type': 'uniform', 'normalize_type': 'tree', 'rate_drop': 1.5890545582941187e-06, 'skip_drop': 0.00016048809963100828}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.75343
[1]	validation-auc:0.76048
[2]	validation-auc:0.76011
[3]	validation-auc:0.76072
[4]	validation-auc:0.77794
[5]	validation-auc:0.77782
[6]	validation-auc:0.77537
[7]	validation-auc:0.76109
[8]	validation-auc:0.76422
[9]	validation-auc:0.76177










    






[I 2020-08-24 17:08:56,403] Finished trial#59 with value: 0.8905109489051095 with parameters: {'booster': 'dart', 'lambda': 0.09178790177803117, 'alpha': 0.1639246755294126, 'max_depth': 5, 'eta': 0.015519011437341223, 'gamma': 0.07933222314454821, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'tree', 'rate_drop': 0.005413222549116395, 'skip_drop': 1.5962380112762536e-06}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.74816
[1]	validation-auc:0.76875
[2]	validation-auc:0.78689
[3]	validation-auc:0.83627
[4]	validation-auc:0.85300
[5]	validation-auc:0.86177
[6]	validation-auc:0.84933
[7]	validation-auc:0.84976
[8]	validation-auc:0.84976
[9]	validation-auc:0.84436










    






[I 2020-08-24 17:08:56,792] Finished trial#60 with value: 0.9051094890510949 with parameters: {'booster': 'gbtree', 'lambda': 0.014119336942422669, 'alpha': 0.0006437426994185446, 'max_depth': 6, 'eta': 0.2913801863803938, 'gamma': 0.18191845539909274, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.71422
[1]	validation-auc:0.73922
[2]	validation-auc:0.73701
[3]	validation-auc:0.74853
[4]	validation-auc:0.74804
[5]	validation-auc:0.75674
[6]	validation-auc:0.77206
[7]	validation-auc:0.75190
[8]	validation-auc:0.75741
[9]	validation-auc:0.77071










    






[I 2020-08-24 17:08:57,071] Finished trial#61 with value: 0.8832116788321168 with parameters: {'booster': 'gbtree', 'lambda': 0.01802095148935071, 'alpha': 0.0006442679387811786, 'max_depth': 7, 'eta': 0.0037612043627880862, 'gamma': 0.0021479833977203467, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.82555
[1]	validation-auc:0.77561
[2]	validation-auc:0.77512
[3]	validation-auc:0.78591
[4]	validation-auc:0.79816
[5]	validation-auc:0.80502
[6]	validation-auc:0.81838
[7]	validation-auc:0.81734
[8]	validation-auc:0.82310
[9]	validation-auc:0.81342










    






[I 2020-08-24 17:08:57,450] Finished trial#62 with value: 0.9051094890510949 with parameters: {'booster': 'gbtree', 'lambda': 0.05017701798097766, 'alpha': 0.007959056875245306, 'max_depth': 4, 'eta': 0.13491311441147724, 'gamma': 0.1778072661463803, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.77469
[1]	validation-auc:0.77567
[2]	validation-auc:0.75257
[3]	validation-auc:0.77004
[4]	validation-auc:0.76771
[5]	validation-auc:0.77837
[6]	validation-auc:0.78523
[7]	validation-auc:0.79626
[8]	validation-auc:0.79804
[9]	validation-auc:0.80147










    






[I 2020-08-24 17:08:57,831] Finished trial#63 with value: 0.8905109489051095 with parameters: {'booster': 'dart', 'lambda': 0.0918815650172444, 'alpha': 0.2565871494523233, 'max_depth': 3, 'eta': 0.10655861911502067, 'gamma': 0.00017208317311268446, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'tree', 'rate_drop': 2.634040817709064e-05, 'skip_drop': 2.5734769178748764e-07}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.76011
[1]	validation-auc:0.79700
[2]	validation-auc:0.80227
[3]	validation-auc:0.79369
[4]	validation-auc:0.82843
[5]	validation-auc:0.81060
[6]	validation-auc:0.78995
[7]	validation-auc:0.79069
[8]	validation-auc:0.79877
[9]	validation-auc:0.79804










    






[I 2020-08-24 17:08:58,090] Finished trial#64 with value: 0.9087591240875912 with parameters: {'booster': 'gbtree', 'lambda': 0.18354712186254957, 'alpha': 0.004448453164245582, 'max_depth': 6, 'eta': 0.3441756080190734, 'gamma': 0.44973542096394264, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.71422
[1]	validation-auc:0.73707
[2]	validation-auc:0.74559
[3]	validation-auc:0.75748
[4]	validation-auc:0.76630
[5]	validation-auc:0.73725
[6]	validation-auc:0.74914
[7]	validation-auc:0.75552
[8]	validation-auc:0.76667
[9]	validation-auc:0.76360










    






[I 2020-08-24 17:08:58,363] Finished trial#65 with value: 0.8759124087591241 with parameters: {'booster': 'gbtree', 'lambda': 0.0016305919416364414, 'alpha': 0.011361262454189306, 'max_depth': 7, 'eta': 0.8083533721795729, 'gamma': 0.49448650038797637, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.76011
[1]	validation-auc:0.76752
[2]	validation-auc:0.76532
[3]	validation-auc:0.74841
[4]	validation-auc:0.74485
[5]	validation-auc:0.74130
[6]	validation-auc:0.78866
[7]	validation-auc:0.81225
[8]	validation-auc:0.80833
[9]	validation-auc:0.82549










    






[I 2020-08-24 17:08:58,669] Finished trial#66 with value: 0.9014598540145985 with parameters: {'booster': 'gbtree', 'lambda': 0.20689261736774212, 'alpha': 0.03643181100681845, 'max_depth': 6, 'eta': 0.03270833220082316, 'gamma': 0.01372644567498579, 'grow_policy': 'lossguide'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.73174
[1]	validation-auc:0.73823
[2]	validation-auc:0.77163
[3]	validation-auc:0.79118
[4]	validation-auc:0.80276
[5]	validation-auc:0.81017
[6]	validation-auc:0.82292
[7]	validation-auc:0.83174
[8]	validation-auc:0.82966
[9]	validation-auc:0.82414










    






[I 2020-08-24 17:08:58,981] Finished trial#67 with value: 0.9014598540145985 with parameters: {'booster': 'gbtree', 'lambda': 0.012227790653089996, 'alpha': 0.005663399230046192, 'max_depth': 8, 'eta': 0.25169368951637133, 'gamma': 0.8926502018654568, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.82555
[1]	validation-auc:0.77439
[2]	validation-auc:0.79246
[3]	validation-auc:0.81661
[4]	validation-auc:0.80643
[5]	validation-auc:0.80067
[6]	validation-auc:0.81091
[7]	validation-auc:0.81630
[8]	validation-auc:0.82488
[9]	validation-auc:0.81998










    






[I 2020-08-24 17:08:59,248] Finished trial#68 with value: 0.9014598540145985 with parameters: {'booster': 'gbtree', 'lambda': 0.0332288256929674, 'alpha': 0.00012174716811289859, 'max_depth': 4, 'eta': 0.0971609068355057, 'gamma': 0.1357173522813667, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.74816
[1]	validation-auc:0.79130
[2]	validation-auc:0.83045
[3]	validation-auc:0.82604
[4]	validation-auc:0.81887
[5]	validation-auc:0.83174
[6]	validation-auc:0.83370
[7]	validation-auc:0.81728
[8]	validation-auc:0.81201
[9]	validation-auc:0.79877










    






[I 2020-08-24 17:08:59,510] Finished trial#69 with value: 0.9087591240875912 with parameters: {'booster': 'gbtree', 'lambda': 0.005914763426314582, 'alpha': 0.002500881444722606, 'max_depth': 6, 'eta': 0.33539302740763777, 'gamma': 0.22845235538460468, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.74816
[1]	validation-auc:0.79718
[2]	validation-auc:0.85055
[3]	validation-auc:0.84044
[4]	validation-auc:0.83940
[5]	validation-auc:0.81697
[6]	validation-auc:0.83493
[7]	validation-auc:0.83922
[8]	validation-auc:0.83971
[9]	validation-auc:0.83315










    






[I 2020-08-24 17:08:59,936] Finished trial#70 with value: 0.8978102189781022 with parameters: {'booster': 'gbtree', 'lambda': 0.007689988603205567, 'alpha': 0.0022900108156415585, 'max_depth': 6, 'eta': 0.17490857956849573, 'gamma': 0.6170471558404279, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.79075
[1]	validation-auc:0.78958
[2]	validation-auc:0.83015
[3]	validation-auc:0.81985
[4]	validation-auc:0.82286
[5]	validation-auc:0.79657
[6]	validation-auc:0.80533
[7]	validation-auc:0.79877
[8]	validation-auc:0.80257
[9]	validation-auc:0.81287










    






[I 2020-08-24 17:09:00,196] Finished trial#71 with value: 0.8905109489051095 with parameters: {'booster': 'dart', 'lambda': 0.4097199025031064, 'alpha': 0.018726383119091146, 'max_depth': 4, 'eta': 0.06964711491807214, 'gamma': 0.061649641490637674, 'grow_policy': 'depthwise', 'sample_type': 'weighted', 'normalize_type': 'tree', 'rate_drop': 2.4008489397391495e-06, 'skip_drop': 3.675355760108881e-08}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.76667
[1]	validation-auc:0.75527
[2]	validation-auc:0.73744
[3]	validation-auc:0.77420
[4]	validation-auc:0.76219
[5]	validation-auc:0.77855
[6]	validation-auc:0.78125
[7]	validation-auc:0.78701
[8]	validation-auc:0.78787
[9]	validation-auc:0.78186










    






[I 2020-08-24 17:09:00,465] Finished trial#72 with value: 0.9124087591240876 with parameters: {'booster': 'gbtree', 'lambda': 0.1387961702310067, 'alpha': 0.00362729469430641, 'max_depth': 8, 'eta': 0.3182818980438705, 'gamma': 0.30571131800910145, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.73174
[1]	validation-auc:0.75741
[2]	validation-auc:0.73756
[3]	validation-auc:0.80172
[4]	validation-auc:0.80686
[5]	validation-auc:0.77059
[6]	validation-auc:0.76667
[7]	validation-auc:0.77647
[8]	validation-auc:0.80000
[9]	validation-auc:0.81311










    






[I 2020-08-24 17:09:00,755] Finished trial#73 with value: 0.9124087591240876 with parameters: {'booster': 'gbtree', 'lambda': 0.02010097502674613, 'alpha': 0.0005691045042325512, 'max_depth': 8, 'eta': 0.41314266512198083, 'gamma': 0.9488520010466092, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.76667
[1]	validation-auc:0.78817
[2]	validation-auc:0.79908
[3]	validation-auc:0.80748
[4]	validation-auc:0.79154
[5]	validation-auc:0.78248
[6]	validation-auc:0.79718
[7]	validation-auc:0.81250
[8]	validation-auc:0.81556
[9]	validation-auc:0.82096










    






[I 2020-08-24 17:09:01,109] Finished trial#74 with value: 0.9014598540145985 with parameters: {'booster': 'gbtree', 'lambda': 0.13465057599918318, 'alpha': 0.0012276741001459673, 'max_depth': 8, 'eta': 0.5180446114902452, 'gamma': 0.30346631980565186, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.73762
[1]	validation-auc:0.76814
[2]	validation-auc:0.78039
[3]	validation-auc:0.82623
[4]	validation-auc:0.79969
[5]	validation-auc:0.79277
[6]	validation-auc:0.80466
[7]	validation-auc:0.79277
[8]	validation-auc:0.80012
[9]	validation-auc:0.80760










    






[I 2020-08-24 17:09:01,518] Finished trial#75 with value: 0.8868613138686131 with parameters: {'booster': 'gbtree', 'lambda': 0.053572165416924715, 'alpha': 0.0034631998235966246, 'max_depth': 9, 'eta': 0.44963911480535146, 'gamma': 0.9803186050982433, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.77984
[1]	validation-auc:0.75490
[2]	validation-auc:0.70208
[3]	validation-auc:0.74559
[4]	validation-auc:0.75098
[5]	validation-auc:0.77071
[6]	validation-auc:0.76422
[7]	validation-auc:0.78468
[8]	validation-auc:0.79069
[9]	validation-auc:0.78640










    






[I 2020-08-24 17:09:01,878] Finished trial#76 with value: 0.8978102189781022 with parameters: {'booster': 'gbtree', 'lambda': 0.2143746909980769, 'alpha': 0.002372246600067276, 'max_depth': 8, 'eta': 0.2955639524674893, 'gamma': 0.5663704066824681, 'grow_policy': 'lossguide'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.80190
[1]	validation-auc:0.77463
[2]	validation-auc:0.80748
[3]	validation-auc:0.83168
[4]	validation-auc:0.84859
[5]	validation-auc:0.82812
[6]	validation-auc:0.83499
[7]	validation-auc:0.83977
[8]	validation-auc:0.83327
[9]	validation-auc:0.83499










    






[I 2020-08-24 17:09:02,171] Finished trial#77 with value: 0.8905109489051095 with parameters: {'booster': 'gbtree', 'lambda': 0.0053631327541176855, 'alpha': 0.00038245461715675805, 'max_depth': 9, 'eta': 0.9399429412139798, 'gamma': 0.2519579910008738, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.77390
[1]	validation-auc:0.72635
[2]	validation-auc:0.78303
[3]	validation-auc:0.80809
[4]	validation-auc:0.83113
[5]	validation-auc:0.83456
[6]	validation-auc:0.83542
[7]	validation-auc:0.82843
[8]	validation-auc:0.82635
[9]	validation-auc:0.82733










    






[I 2020-08-24 17:09:02,456] Finished trial#78 with value: 0.8978102189781022 with parameters: {'booster': 'gbtree', 'lambda': 0.00022631497878447166, 'alpha': 0.00012160952061989604, 'max_depth': 8, 'eta': 0.9863956941688855, 'gamma': 0.10088634273197924, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.57316
[1]	validation-auc:0.73872
[2]	validation-auc:0.79608
[3]	validation-auc:0.80368
[4]	validation-auc:0.80600
[5]	validation-auc:0.80858
[6]	validation-auc:0.80870
[7]	validation-auc:0.80760
[8]	validation-auc:0.80821
[9]	validation-auc:0.80649










    






[I 2020-08-24 17:09:02,675] Finished trial#79 with value: 0.8759124087591241 with parameters: {'booster': 'gblinear', 'lambda': 0.021387962969461546, 'alpha': 0.00020620585637643404}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.69945
[1]	validation-auc:0.71808
[2]	validation-auc:0.75827
[3]	validation-auc:0.82616
[4]	validation-auc:0.84424
[5]	validation-auc:0.85698
[6]	validation-auc:0.85895
[7]	validation-auc:0.85123
[8]	validation-auc:0.84301
[9]	validation-auc:0.84657










    






[I 2020-08-24 17:09:03,071] Finished trial#80 with value: 0.9087591240875912 with parameters: {'booster': 'gbtree', 'lambda': 2.574239716067691e-06, 'alpha': 0.0009385435189772736, 'max_depth': 7, 'eta': 0.1514589059750527, 'gamma': 0.050453324736496344, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.69945
[1]	validation-auc:0.67690
[2]	validation-auc:0.73474
[3]	validation-auc:0.74926
[4]	validation-auc:0.78781
[5]	validation-auc:0.79075
[6]	validation-auc:0.79418
[7]	validation-auc:0.80931
[8]	validation-auc:0.79951
[9]	validation-auc:0.79706










    






[I 2020-08-24 17:09:03,420] Finished trial#81 with value: 0.8941605839416058 with parameters: {'booster': 'gbtree', 'lambda': 5.534600079123594e-07, 'alpha': 0.00038945989417794717, 'max_depth': 7, 'eta': 0.05586998822327946, 'gamma': 0.03866390906430895, 'grow_policy': 'depthwise'}. Best is trial#29 with value: 0.9124087591240876.










    






[0]	validation-auc:0.69945
[1]	validation-auc:0.67592
[2]	validation-auc:0.73003
[3]	validation-auc:0.79804
[4]	validation-auc:0.78683
[5]	validation-auc:0.82230
[6]	validation-auc:0.82120
[7]	validation-auc:0.84105
[8]	validation-auc:0.83946
[9]	validation-auc:0.83431










    






[I 2020-08-24 17:09:03,691] Finished trial#82 with value: 0.916058394160584 with parameters: {'booster': 'gbtree', 'lambda': 2.9016833274930767e-05, 'alpha': 0.0008461825479637201, 'max_depth': 7, 'eta': 0.13819724043277082, 'gamma': 0.10726812017919862, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.69945
[1]	validation-auc:0.71844
[2]	validation-auc:0.74038
[3]	validation-auc:0.78885
[4]	validation-auc:0.78456
[5]	validation-auc:0.80355
[6]	validation-auc:0.80784
[7]	validation-auc:0.81140
[8]	validation-auc:0.80245
[9]	validation-auc:0.81875










    






[I 2020-08-24 17:09:03,970] Finished trial#83 with value: 0.9087591240875912 with parameters: {'booster': 'gbtree', 'lambda': 2.1148518737195673e-06, 'alpha': 0.000956751804554467, 'max_depth': 7, 'eta': 0.1693873032761029, 'gamma': 0.08974363069212565, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.69945
[1]	validation-auc:0.72261
[2]	validation-auc:0.75411
[3]	validation-auc:0.81256
[4]	validation-auc:0.81985
[5]	validation-auc:0.83223
[6]	validation-auc:0.83015
[7]	validation-auc:0.82880
[8]	validation-auc:0.83468
[9]	validation-auc:0.83137










    






[I 2020-08-24 17:09:04,337] Finished trial#84 with value: 0.8941605839416058 with parameters: {'booster': 'gbtree', 'lambda': 4.833250101177598e-06, 'alpha': 0.000868941302743293, 'max_depth': 7, 'eta': 0.39324963284977354, 'gamma': 0.09341402312211876, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.77390
[1]	validation-auc:0.75466
[2]	validation-auc:0.74387
[3]	validation-auc:0.81366
[4]	validation-auc:0.82451
[5]	validation-auc:0.82433
[6]	validation-auc:0.81918
[7]	validation-auc:0.81765
[8]	validation-auc:0.82328
[9]	validation-auc:0.83860










    






[I 2020-08-24 17:09:04,695] Finished trial#85 with value: 0.9051094890510949 with parameters: {'booster': 'gbtree', 'lambda': 3.6654215655370234e-06, 'alpha': 5.1835762278677125e-05, 'max_depth': 8, 'eta': 0.1454644705990994, 'gamma': 0.007970843724365266, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.69945
[1]	validation-auc:0.71857
[2]	validation-auc:0.74308
[3]	validation-auc:0.78891
[4]	validation-auc:0.80558
[5]	validation-auc:0.81275
[6]	validation-auc:0.80294
[7]	validation-auc:0.83358
[8]	validation-auc:0.83603
[9]	validation-auc:0.84240










    






[I 2020-08-24 17:09:05,068] Finished trial#86 with value: 0.9014598540145985 with parameters: {'booster': 'gbtree', 'lambda': 1.7961048280585732e-06, 'alpha': 0.0014150125101717386, 'max_depth': 7, 'eta': 0.17385193714398217, 'gamma': 0.05267299119971756, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.71422
[1]	validation-auc:0.73787
[2]	validation-auc:0.77861
[3]	validation-auc:0.78241
[4]	validation-auc:0.77586
[5]	validation-auc:0.76409
[6]	validation-auc:0.76348
[7]	validation-auc:0.76489
[8]	validation-auc:0.77874
[9]	validation-auc:0.78971










    






[I 2020-08-24 17:09:05,532] Finished trial#87 with value: 0.8905109489051095 with parameters: {'booster': 'gbtree', 'lambda': 1.176517237571868e-05, 'alpha': 0.004814122060720835, 'max_depth': 7, 'eta': 0.01750619372463899, 'gamma': 0.2844711883227655, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.77390
[1]	validation-auc:0.79736
[2]	validation-auc:0.80141
[3]	validation-auc:0.81023
[4]	validation-auc:0.79461
[5]	validation-auc:0.80331
[6]	validation-auc:0.80625
[7]	validation-auc:0.82390
[8]	validation-auc:0.83137
[9]	validation-auc:0.82218










    






[I 2020-08-24 17:09:05,826] Finished trial#88 with value: 0.9087591240875912 with parameters: {'booster': 'gbtree', 'lambda': 5.762686113859439e-05, 'alpha': 0.0008159785982816633, 'max_depth': 8, 'eta': 0.5209674943297983, 'gamma': 0.3684078350768504, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.72806
[1]	validation-auc:0.76569
[2]	validation-auc:0.77757
[3]	validation-auc:0.80441
[4]	validation-auc:0.83578
[5]	validation-auc:0.82855
[6]	validation-auc:0.82402
[7]	validation-auc:0.82249
[8]	validation-auc:0.82243
[9]	validation-auc:0.83027










    






[I 2020-08-24 17:09:06,130] Finished trial#89 with value: 0.9087591240875912 with parameters: {'booster': 'gbtree', 'lambda': 5.2985274375988565e-05, 'alpha': 8.271366753157e-05, 'max_depth': 8, 'eta': 0.4918024435665885, 'gamma': 0.9560139652435061, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.71415
[1]	validation-auc:0.73235
[2]	validation-auc:0.79142
[3]	validation-auc:0.80098
[4]	validation-auc:0.80245
[5]	validation-auc:0.81164
[6]	validation-auc:0.80674
[7]	validation-auc:0.81336
[8]	validation-auc:0.80993
[9]	validation-auc:0.80221










    






[I 2020-08-24 17:09:06,443] Finished trial#90 with value: 0.8941605839416058 with parameters: {'booster': 'gbtree', 'lambda': 4.9505926120230405e-05, 'alpha': 3.1654514852378665e-05, 'max_depth': 8, 'eta': 0.6421419312013555, 'gamma': 0.863448332591534, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.78952
[1]	validation-auc:0.73707
[2]	validation-auc:0.72776
[3]	validation-auc:0.79455
[4]	validation-auc:0.78352
[5]	validation-auc:0.79271
[6]	validation-auc:0.78928
[7]	validation-auc:0.77396
[8]	validation-auc:0.77904
[9]	validation-auc:0.79222










    






[I 2020-08-24 17:09:06,744] Finished trial#91 with value: 0.9014598540145985 with parameters: {'booster': 'gbtree', 'lambda': 2.0999826701144247e-06, 'alpha': 0.000813097746653807, 'max_depth': 9, 'eta': 0.08054087767295073, 'gamma': 0.5556903734938329, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.77390
[1]	validation-auc:0.74210
[2]	validation-auc:0.79651
[3]	validation-auc:0.81434
[4]	validation-auc:0.81409
[5]	validation-auc:0.80711
[6]	validation-auc:0.81262
[7]	validation-auc:0.82757
[8]	validation-auc:0.81985
[9]	validation-auc:0.82684










    






[I 2020-08-24 17:09:07,087] Finished trial#92 with value: 0.9014598540145985 with parameters: {'booster': 'gbtree', 'lambda': 4.143567746835493e-05, 'alpha': 8.862553422690998e-05, 'max_depth': 8, 'eta': 0.22613300389114818, 'gamma': 0.24289359811751682, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.78952
[1]	validation-auc:0.75582
[2]	validation-auc:0.76489
[3]	validation-auc:0.79828
[4]	validation-auc:0.78965
[5]	validation-auc:0.79810
[6]	validation-auc:0.79871
[7]	validation-auc:0.79920
[8]	validation-auc:0.81483
[9]	validation-auc:0.82010










    






[I 2020-08-24 17:09:07,405] Finished trial#93 with value: 0.9014598540145985 with parameters: {'booster': 'gbtree', 'lambda': 0.00015824556917259371, 'alpha': 0.0002510689892952314, 'max_depth': 9, 'eta': 0.1363658018466474, 'gamma': 0.10708342992165698, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.72806
[1]	validation-auc:0.78284
[2]	validation-auc:0.75974
[3]	validation-auc:0.80031
[4]	validation-auc:0.81875
[5]	validation-auc:0.82953
[6]	validation-auc:0.81777
[7]	validation-auc:0.80944
[8]	validation-auc:0.81213
[9]	validation-auc:0.81336










    






[I 2020-08-24 17:09:07,705] Finished trial#94 with value: 0.9014598540145985 with parameters: {'booster': 'gbtree', 'lambda': 6.910474230284724e-05, 'alpha': 0.00047515625643507626, 'max_depth': 8, 'eta': 0.5433348950396378, 'gamma': 0.9362716542259443, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.77623
[1]	validation-auc:0.73425
[2]	validation-auc:0.74994
[3]	validation-auc:0.81440
[4]	validation-auc:0.83198
[5]	validation-auc:0.83186
[6]	validation-auc:0.83370
[7]	validation-auc:0.82653
[8]	validation-auc:0.83548
[9]	validation-auc:0.82739










    






[I 2020-08-24 17:09:08,123] Finished trial#95 with value: 0.8941605839416058 with parameters: {'booster': 'gbtree', 'lambda': 1.9860256569260252e-05, 'alpha': 0.0001624820690723915, 'max_depth': 8, 'eta': 0.061023643210418005, 'gamma': 0.396643161936805, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.69749
[1]	validation-auc:0.73560
[2]	validation-auc:0.74467
[3]	validation-auc:0.77929
[4]	validation-auc:0.76869
[5]	validation-auc:0.76146
[6]	validation-auc:0.76403
[7]	validation-auc:0.75993
[8]	validation-auc:0.76023
[9]	validation-auc:0.75619










    






[I 2020-08-24 17:09:08,521] Finished trial#96 with value: 0.8905109489051095 with parameters: {'booster': 'gbtree', 'lambda': 1.4496551434911967e-05, 'alpha': 1.3130528646785772e-05, 'max_depth': 7, 'eta': 0.011225968380197227, 'gamma': 0.027228556784170677, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.78952
[1]	validation-auc:0.78836
[2]	validation-auc:0.76109
[3]	validation-auc:0.74884
[4]	validation-auc:0.74614
[5]	validation-auc:0.74222
[6]	validation-auc:0.75809
[7]	validation-auc:0.75417
[8]	validation-auc:0.77678
[9]	validation-auc:0.79485










    






[I 2020-08-24 17:09:08,966] Finished trial#97 with value: 0.8905109489051095 with parameters: {'booster': 'gbtree', 'lambda': 9.215505069743917e-07, 'alpha': 0.0018544456682796965, 'max_depth': 9, 'eta': 0.023649624642935668, 'gamma': 0.06806070965163447, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.71422
[1]	validation-auc:0.73609
[2]	validation-auc:0.79142
[3]	validation-auc:0.79939
[4]	validation-auc:0.80276
[5]	validation-auc:0.81710
[6]	validation-auc:0.82623
[7]	validation-auc:0.82439
[8]	validation-auc:0.82917
[9]	validation-auc:0.83885










    






[I 2020-08-24 17:09:09,243] Finished trial#98 with value: 0.9087591240875912 with parameters: {'booster': 'gbtree', 'lambda': 2.9447416881962137e-07, 'alpha': 0.003044365789939651, 'max_depth': 7, 'eta': 0.23151318048366323, 'gamma': 0.4151197997892498, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






[0]	validation-auc:0.74816
[1]	validation-auc:0.81213
[2]	validation-auc:0.83266
[3]	validation-auc:0.83198
[4]	validation-auc:0.81385
[5]	validation-auc:0.82206
[6]	validation-auc:0.82139
[7]	validation-auc:0.82978
[8]	validation-auc:0.81189
[9]	validation-auc:0.81262










    






[I 2020-08-24 17:09:09,525] Finished trial#99 with value: 0.8868613138686131 with parameters: {'booster': 'gbtree', 'lambda': 0.0003504185843994702, 'alpha': 0.0033171192139125646, 'max_depth': 6, 'eta': 0.3573017686948089, 'gamma': 0.12050776961234781, 'grow_policy': 'depthwise'}. Best is trial#82 with value: 0.916058394160584.










    






FrozenTrial(number=82, value=0.916058394160584, datetime_start=datetime.datetime(2020, 8, 24, 17, 9, 3, 432200), datetime_complete=datetime.datetime(2020, 8, 24, 17, 9, 3, 691462), params={'booster': 'gbtree', 'lambda': 2.9016833274930767e-05, 'alpha': 0.0008461825479637201, 'max_depth': 7, 'eta': 0.13819724043277082, 'gamma': 0.10726812017919862, 'grow_policy': 'depthwise'}, distributions={'booster': CategoricalDistribution(choices=('gbtree', 'gblinear', 'dart')), 'lambda': LogUniformDistribution(high=1.0, low=1e-08), 'alpha': LogUniformDistribution(high=1.0, low=1e-08), 'max_depth': IntUniformDistribution(high=9, low=1, step=1), 'eta': LogUniformDistribution(high=1.0, low=1e-08), 'gamma': LogUniformDistribution(high=1.0, low=1e-08), 'grow_policy': CategoricalDistribution(choices=('depthwise', 'lossguide'))}, user_attrs={}, system_attrs={}, intermediate_values={0: 0.699449, 1: 0.675919, 2: 0.730025, 3: 0.798039, 4: 0.786826, 5: 0.822304, 6: 0.821201, 7: 0.841054, 8: 0.839461, 9: 0.834314}, trial_id=82, state=TrialState.COMPLETE)



















In [19]:



    


study_v2.best_params



    















    
Out[19]:








{'booster': 'gbtree',
 'lambda': 2.9016833274930767e-05,
 'alpha': 0.0008461825479637201,
 'max_depth': 7,
 'eta': 0.13819724043277082,
 'gamma': 0.10726812017919862,
 'grow_policy': 'depthwise'}



















In [20]:



    


# Criando o modelo
modelo_xgb_v4 = xgb.XGBClassifier(**study_v2.best_params)

# Treinando o modelo
modelo_xgb_v4.fit(X_train, y_train)

# Fazendo previsões
xgb_y_pred = modelo_xgb_v4.predict(X_test)
previsoes = [round(value) for value in xgb_y_pred]

# Avaliando as previsões
xgb_score_v4 = accuracy_score(y_test, previsoes)
print("Acurácia: %.2f%%" % (xgb_score_v4 * 100.0))

# Cria a matriz de confusão
conf_matriz_xgb = confusion_matrix(y_test, xgb_y_pred)
    
# Calcula especificidade e sensibilidade
speci_xgb = conf_matriz_xgb[0,0] / (conf_matriz_xgb[0,0] + conf_matriz_xgb[0,1])
sensi_xgb = conf_matriz_xgb[1,1] / (conf_matriz_xgb[1,0] + conf_matriz_xgb[1,1])

# Print
print('Sensibilidade :', sensi_xgb)
print('Especificidade :', speci_xgb)
print('\n')

# Matriz de Confusão gráfica
sns.set(rc={'figure.figsize':(6, 6)})
sns.heatmap(conf_matriz_xgb, 
            annot = True, 
            fmt = ".0f", 
            linewidths = .5, 
            square = True, 
            cmap = 'RdBu_r')

# Labels e Títulos
plt.ylabel('Label Verdadeiro')
plt.xlabel('Label Previsto')
plt.title('Acurácia: {:.2f}'.format(xgb_score_v4), size = 15)

# Relatório de Classificação
print(classification_report(y_test, xgb_y_pred))



    















    






Acurácia: 94.07%
Sensibilidade : 0.9424778761061947
Especificidade : 0.9390243902439024


              precision    recall  f1-score   support

           0       0.95      0.94      0.94       246
           1       0.93      0.94      0.94       226

    accuracy                           0.94       472
   macro avg       0.94      0.94      0.94       472
weighted avg       0.94      0.94      0.94       472











    




























Parte 07. Avaliação do Desempenho dos Modelos
Nesta fase estarei comparando cada versao do modelo Random Forest















In [22]:



    


# Prepara a lista de resultados
metricas = [(xgb_score_v1),(xgb_score_v2),(xgb_score_v3),(xgb_score_v4)]

# Cria o dataframe
df_metricas = pd.DataFrame(metricas, 
                           columns = ['Accuracy'], 
                           index = ['Versao 01','Versao 02','Versao 03','Versao 04']) 

# Visualiza o resultado
df_metricas



    















    
Out[22]:











  
    

      
      Accuracy
    

  
  
    

      Versao 01
      0.897810
    

    

      Versao 02
      0.930085
    

    

      Versao 03
      0.938559
    

    

      Versao 04
      0.940678
    

  






















In [ ]: